[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:12 dist-test-slave-0f9d krb5kdc[27023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:12 dist-test-slave-0f9d krb5kdc[27023](info): set up 2 sockets
Jun 19 02:49:12 dist-test-slave-0f9d krb5kdc[27023](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:49:13.656260 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:13 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:13.793985 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46571
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:13.894416 27036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:13.894747 27036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:13.894789 27036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:13.896551 27036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:13.896651 27036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:13.896672 27036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:13.896692 27036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:13.896709 27036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:13.898861 27036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:13.899286 27036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:13.899726 27036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:13.905679 27044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.905830 27041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.905833 27042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:13.906091 27036 server_base.cc:1034] running on GCE node
I20260619 02:49:13.906446 27036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:13.906833 27036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:13.908077 27036 hybrid_clock.cc:648] HybridClock initialized: now 1781837353908052 us; error 40 us; skew 500 ppm
Jun 19 02:49:13 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837353, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:13.911657 27036 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:13.912550 27036 webserver.cc:460] Webserver started at http://127.26.98.62:35637/ using document root <none> and password file <none>
I20260619 02:49:13.912976 27036 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:13.913050 27036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:13.913203 27036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:13.914815 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "2116c491a949489da15b85ebc4c64112"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-0f9d"
server_key: "b545b7a91e332603c333e76d45b42d8d"
server_key_iv: "1c65fdb0148a817d441bed09d8f97897"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.915565 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "2116c491a949489da15b85ebc4c64112"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-0f9d"
server_key: "b545b7a91e332603c333e76d45b42d8d"
server_key_iv: "1c65fdb0148a817d441bed09d8f97897"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.917603 27036 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:13.918617 27051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:13.918921 27036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:13.919032 27036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "2116c491a949489da15b85ebc4c64112"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-0f9d"
server_key: "b545b7a91e332603c333e76d45b42d8d"
server_key_iv: "1c65fdb0148a817d441bed09d8f97897"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.919133 27036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:13.932466 27036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:13.936125 27036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:13.936309 27036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:13.941473 27036 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:46571
I20260619 02:49:13.941540 27103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:46571 every 8 connection(s)
I20260619 02:49:13.941990 27036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:13.943272 27104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:13.946733 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: Bootstrap starting.
I20260619 02:49:13.947625 27104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:13.947948 27104 log.cc:824] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:13.949002 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: No bootstrap required, opened a new log
I20260619 02:49:13.951164 27104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } }
I20260619 02:49:13.951390 27104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:13.951427 27104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116c491a949489da15b85ebc4c64112, State: Initialized, Role: FOLLOWER
I20260619 02:49:13.951617 27104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } }
I20260619 02:49:13.951716 27104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:13.951762 27104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:13.951824 27104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:13.952256 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27036
I20260619 02:49:13.952378 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:13.952705 27104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } }
I20260619 02:49:13.952847 27104 leader_election.cc:302] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2116c491a949489da15b85ebc4c64112; no voters:
I20260619 02:49:13.953094 27104 leader_election.cc:288] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:13.953135 27107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:13.953265 27107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [term 1 LEADER]: Becoming Leader. State: Replica: 2116c491a949489da15b85ebc4c64112, State: Running, Role: LEADER
I20260619 02:49:13.953258 27016 external_mini_cluster.cc:1428] Setting key 9f6f9d8334190c29e919cd476f9e07a7
I20260619 02:49:13.953367 27107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } }
I20260619 02:49:13.953459 27104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:13.954007 27108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2116c491a949489da15b85ebc4c64112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } } }
I20260619 02:49:13.954114 27108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:13.954661 27109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2116c491a949489da15b85ebc4c64112. Latest consensus state: current_term: 1 leader_uuid: "2116c491a949489da15b85ebc4c64112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2116c491a949489da15b85ebc4c64112" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46571 } } }
I20260619 02:49:13.954823 27109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112 [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:13.955849 27125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:13.955928 27125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:13.956146 27116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:13.956390 27116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:13.958688 27116 catalog_manager.cc:1347] Generated new cluster ID: e9b8f572582b4543bb337e8581b91992
I20260619 02:49:13.958791 27116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:13 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:13.973872 27116 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:13.974727 27116 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:13.976572 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:13.954469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.26.98.62:46571)
0619 02:49:13.954932 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:13.954942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:13.955063 (+ 121us) server_negotiation.cc:407] Connection header received
0619 02:49:13.957967 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:13.957977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:13.958253 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:13.958505 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:13.961006 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.962740 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.964688 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.965442 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.965608 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:13.971583 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:13.971592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:13.971593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:13.971613 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:13.974161 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:13.974765 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:13.974770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:13.974771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:13.974944 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:13.975224 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:13.975225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:13.975225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:13.975516 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:13.975706 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:13.975986 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:49:13.976153 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":376,"thread_start_us":94,"threads_started":1}
I20260619 02:49:13.979903 27116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2116c491a949489da15b85ebc4c64112: Generated new TSK 0
I20260619 02:49:13.980124 27116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:14.014699 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:14.124745 27130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:14.125090 27130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:14.125126 27130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:14.126847 27130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:14.126997 27130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:14.127055 27130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:14.129863 27130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:14.130352 27130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:14.130702 27130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:14.138598 27136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.138595 27138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.138545 27135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.139402 27130 server_base.cc:1034] running on GCE node
I20260619 02:49:14.139779 27130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:14.140241 27130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:14.141510 27130 hybrid_clock.cc:648] HybridClock initialized: now 1781837354141487 us; error 44 us; skew 500 ppm
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:14.145381 27130 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:14.146057 27130 webserver.cc:460] Webserver started at http://127.26.98.1:34485/ using document root <none> and password file <none>
I20260619 02:49:14.146453 27130 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:14.146546 27130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:14.146727 27130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:14.148420 27130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "24869d638a2041c399525ed71ed60e03"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "d258969a7bbb2c378a2f9b38db8e94bd"
server_key_iv: "6b3f371ab3f7fb9f445bd64f88a096af"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.148957 27130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "24869d638a2041c399525ed71ed60e03"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "d258969a7bbb2c378a2f9b38db8e94bd"
server_key_iv: "6b3f371ab3f7fb9f445bd64f88a096af"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.150986 27130 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:49:14.152347 27145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.152748 27130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:14.152909 27130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "24869d638a2041c399525ed71ed60e03"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "d258969a7bbb2c378a2f9b38db8e94bd"
server_key_iv: "6b3f371ab3f7fb9f445bd64f88a096af"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.153100 27130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:14.201265 27130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:14.205401 27130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:14.205622 27130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:14.205955 27130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:14.206365 27130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:14.206408 27130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.206449 27130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:14.206473 27130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.214758 27130 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:40047
I20260619 02:49:14.214988 27258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:40047 every 8 connection(s)
I20260619 02:49:14.215276 27130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:14.223094 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27130
I20260619 02:49:14.223230 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:14.223500 27016 external_mini_cluster.cc:1428] Setting key f872bcb05191061da005b112f1a4be97
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:14.230397 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.216257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:44811 (local address 127.26.98.62:46571)
0619 02:49:14.216359 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.216362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.216635 (+ 273us) server_negotiation.cc:407] Connection header received
0619 02:49:14.217322 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.217323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.217341 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.217451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.219014 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.219612 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.221533 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.221843 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.221951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.225310 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.225312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.225312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:14.225322 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.227271 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.228040 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.228042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.228043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.228130 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.229303 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.229306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.229306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.229531 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.229657 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.230100 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.230264 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:49:14.230679 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.216167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46571 (local address 127.26.98.1:44811)
0619 02:49:14.216434 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.216513 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.217172 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.217676 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.217682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.218442 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:14.218759 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.218767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.219835 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.219837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.221307 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.221317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.222003 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.222005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.222188 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.222960 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:14.222974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:14.225176 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:14.227443 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.227460 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.227464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.227944 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.228530 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.228531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.228532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.228830 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.229679 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:14.229693 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:14.229918 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.230138 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.230370 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":128,"threads_started":1}
I20260619 02:49:14.231451 27259 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46571
I20260619 02:49:14.231629 27259 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:14.231920 27259 heartbeater.cc:511] Master 127.26.98.62:46571 requested a full tablet report, sending...
I20260619 02:49:14.232764 27068 ts_manager.cc:194] Registered new tserver with Master: 24869d638a2041c399525ed71ed60e03 (127.26.98.1:40047)
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:49:14.233534 27068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:44811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:14.258719 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:14.360603 27264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:14.360948 27264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:14.360985 27264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:14.362607 27264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:14.362674 27264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:14.362712 27264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:14.364758 27264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:14.365048 27264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:14.365295 27264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:14.371627 27270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.371752 27272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.371752 27269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.371793 27264 server_base.cc:1034] running on GCE node
I20260619 02:49:14.372304 27264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:14.372654 27264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:14.373864 27264 hybrid_clock.cc:648] HybridClock initialized: now 1781837354373831 us; error 74 us; skew 500 ppm
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:14.377702 27264 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:14.378288 27264 webserver.cc:460] Webserver started at http://127.26.98.2:46577/ using document root <none> and password file <none>
I20260619 02:49:14.378535 27264 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:14.378592 27264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:14.378732 27264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:14.380074 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "54687724d409426db766969fc12bc967"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "2452db83c3470ce53eeb87d1086e5c23"
server_key_iv: "787156fbd26847c03302c456aa75e19a"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.380470 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "54687724d409426db766969fc12bc967"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "2452db83c3470ce53eeb87d1086e5c23"
server_key_iv: "787156fbd26847c03302c456aa75e19a"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.382328 27264 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:49:14.383440 27279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.383659 27264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:14.383853 27264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "54687724d409426db766969fc12bc967"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "2452db83c3470ce53eeb87d1086e5c23"
server_key_iv: "787156fbd26847c03302c456aa75e19a"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.383955 27264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:14.407980 27264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:14.410998 27264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:14.411202 27264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:14.411520 27264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:14.411944 27264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:14.411991 27264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.412029 27264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:14.412050 27264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.419592 27264 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40873
I20260619 02:49:14.419637 27392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40873 every 8 connection(s)
I20260619 02:49:14.420109 27264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:14.425832 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27264
I20260619 02:49:14.425932 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:14.426105 27016 external_mini_cluster.cc:1428] Setting key 0e78f1a9e96d26cf14c1adfb22447609
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:14.434360 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.421278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:56453 (local address 127.26.98.62:46571)
0619 02:49:14.421344 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.421346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.421742 (+ 396us) server_negotiation.cc:407] Connection header received
0619 02:49:14.422387 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.422388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.422402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.422465 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.423829 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.424621 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.426228 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.426567 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.426664 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.429946 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.429950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.429950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:14.429962 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.431755 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.432711 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.432713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.432714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.432810 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.433283 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.433284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.433285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.433442 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.433512 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.433983 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.434098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:49:14.434571 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.421241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46571 (local address 127.26.98.2:56453)
0619 02:49:14.421543 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.421596 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.422269 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.422542 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.422546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.423327 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:14.423620 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.423628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.424801 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.424803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.426088 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.426093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.426746 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.426748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.427003 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.427873 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:14.427886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:14.429750 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:14.431900 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.431921 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.431926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.432556 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.432902 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.432903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.432904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.433168 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.433521 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:14.433530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:14.433791 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.433948 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.434253 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":144,"threads_started":1}
I20260619 02:49:14.435526 27393 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46571
I20260619 02:49:14.435693 27393 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:14.436008 27393 heartbeater.cc:511] Master 127.26.98.62:46571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:49:14.436676 27068 ts_manager.cc:194] Registered new tserver with Master: 54687724d409426db766969fc12bc967 (127.26.98.2:40873)
I20260619 02:49:14.437132 27068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:56453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:14.459808 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:14.559890 27398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:14.560221 27398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:14.560268 27398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:14.562386 27398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:14.562522 27398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:14.562582 27398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:14.565250 27398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:14.565716 27398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:14.566061 27398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:14.574259 27406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.574249 27403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.574311 27398 server_base.cc:1034] running on GCE node
W20260619 02:49:14.574266 27404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.575001 27398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:14.575368 27398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:14.576555 27398 hybrid_clock.cc:648] HybridClock initialized: now 1781837354576501 us; error 82 us; skew 500 ppm
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:14.579906 27398 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:14.580477 27398 webserver.cc:460] Webserver started at http://127.26.98.3:36597/ using document root <none> and password file <none>
I20260619 02:49:14.580722 27398 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:14.580775 27398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:14.580883 27398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:14.581974 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "fef16f704895472b90b1eafa697b1ab2"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "3a69bdfd09434d58fec922e2929c852f"
server_key_iv: "e4b8a69f7e3085daef7b6d2ba0388edc"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.582729 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "fef16f704895472b90b1eafa697b1ab2"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "3a69bdfd09434d58fec922e2929c852f"
server_key_iv: "e4b8a69f7e3085daef7b6d2ba0388edc"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.584676 27398 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 02:49:14.585911 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.586253 27398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:14.586373 27398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "fef16f704895472b90b1eafa697b1ab2"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-0f9d"
server_key: "3a69bdfd09434d58fec922e2929c852f"
server_key_iv: "e4b8a69f7e3085daef7b6d2ba0388edc"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.586525 27398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:14.608332 27398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:14.611560 27398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:14.611773 27398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:14.612078 27398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:14.612478 27398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:14.612514 27398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.612540 27398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:14.612550 27398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.620530 27398 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:45961
I20260619 02:49:14.620577 27526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:45961 every 8 connection(s)
I20260619 02:49:14.620992 27398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:14.627475 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27398
I20260619 02:49:14.627580 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:14.627774 27016 external_mini_cluster.cc:1428] Setting key 104397d723696772d4e308c8b8b6af05
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:14.633201 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.621919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46571 (local address 127.26.98.3:42813)
0619 02:49:14.622146 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.622194 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.622800 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.623175 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.623179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.623867 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:14.624116 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.624121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.624946 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.624947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.625883 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.625887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.626323 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.626325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.626481 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.627176 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:14.627187 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:14.629460 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:14.631382 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.631393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.631397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.631739 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.632018 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:14.632019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:14.632019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:14.632150 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:14.632441 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:14.632448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:14.632627 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.632763 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.632967 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
I20260619 02:49:14.633232 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.621904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:42813 (local address 127.26.98.62:46571)
0619 02:49:14.622001 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.622004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.622294 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:49:14.623009 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.623010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.623026 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.623096 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.624226 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.624837 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.626001 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.626204 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.626274 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.629595 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.629597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.629597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:14.629606 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.631264 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.631830 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.631832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.631832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.631934 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.632213 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.632213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.632214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.632356 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.632419 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.633054 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.633153 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:49:14.634068 27527 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46571
I20260619 02:49:14.634213 27527 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:14.634410 27527 heartbeater.cc:511] Master 127.26.98.62:46571 requested a full tablet report, sending...
I20260619 02:49:14.635061 27068 ts_manager.cc:194] Registered new tserver with Master: fef16f704895472b90b1eafa697b1ab2 (127.26.98.3:45961)
I20260619 02:49:14.635492 27068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:42813
I20260619 02:49:14.641047 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:14.670948 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.661084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.26.98.1:40047)
0619 02:49:14.661310 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.661316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.661333 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:14.661369 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.661371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.661390 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.661506 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.662231 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.663168 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.664015 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.664413 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.664486 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.667271 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.667284 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.667285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:14.667295 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.669172 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.669707 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.669711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.669712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.669806 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.670096 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.670097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.670098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.670328 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.670482 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.670677 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.670790 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
W20260619 02:49:14.671391 27173 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50718
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:14.699954 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.690971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.26.98.1:40047)
0619 02:49:14.691041 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.691044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.691249 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:49:14.691341 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.691343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.691362 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.691421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.692240 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.692804 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.693635 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.693872 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.693943 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.696776 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.696779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.696779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:14.696789 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.698637 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.699116 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.699118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.699118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.699180 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:14.699386 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:14.699387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:14.699387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:14.699509 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.699638 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.699721 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.699842 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:49:14.700690 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27130
I20260619 02:49:14.706760 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27264
I20260619 02:49:14.711850 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27398
I20260619 02:49:14.717133 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27036
2026-06-19T02:49:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2084 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27541](info): set up 2 sockets
Jun 19 02:49:14 dist-test-slave-0f9d krb5kdc[27541](info): commencing operation
krb5kdc: starting...
W20260619 02:49:15.722672 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:15 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:15.862380 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35181
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:15.961908 27554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:15.962250 27554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:15.962283 27554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:15.964042 27554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:15.964149 27554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:15.964165 27554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:15.964177 27554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:15.964190 27554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:15.966280 27554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:15.966636 27554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:15.966939 27554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:15.972769 27562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:15.972769 27559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:15.972782 27560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:15.973093 27554 server_base.cc:1034] running on GCE node
I20260619 02:49:15.973414 27554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:15.973767 27554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:15.974951 27554 hybrid_clock.cc:648] HybridClock initialized: now 1781837355974929 us; error 41 us; skew 500 ppm
Jun 19 02:49:15 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837355, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:15.978009 27554 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:15.978552 27554 webserver.cc:460] Webserver started at http://127.26.98.62:44313/ using document root <none> and password file <none>
I20260619 02:49:15.978789 27554 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:15.978843 27554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:15.979024 27554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:15.980347 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d"
format_stamp: "Formatted at 2026-06-19 02:49:15 on dist-test-slave-0f9d"
server_key: "7b9058681ed33e6eb6d71258b489fab0"
server_key_iv: "ce0305261f43c1ae147fdc59dd1a7564"
server_key_version: "encryptionkey@0"
I20260619 02:49:15.980757 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d"
format_stamp: "Formatted at 2026-06-19 02:49:15 on dist-test-slave-0f9d"
server_key: "7b9058681ed33e6eb6d71258b489fab0"
server_key_iv: "ce0305261f43c1ae147fdc59dd1a7564"
server_key_version: "encryptionkey@0"
I20260619 02:49:15.982482 27554 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:15.983603 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:15.983804 27554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:15.983922 27554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d"
format_stamp: "Formatted at 2026-06-19 02:49:15 on dist-test-slave-0f9d"
server_key: "7b9058681ed33e6eb6d71258b489fab0"
server_key_iv: "ce0305261f43c1ae147fdc59dd1a7564"
server_key_version: "encryptionkey@0"
I20260619 02:49:15.984015 27554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:15.998695 27554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:16.001813 27554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:16.002003 27554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:16.007031 27554 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35181
I20260619 02:49:16.007086 27621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35181 every 8 connection(s)
I20260619 02:49:16.007512 27554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:16.008615 27622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:16.010700 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27554
I20260619 02:49:16.010814 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:16.011025 27016 external_mini_cluster.cc:1428] Setting key 51ba724234f914449cfd38729ea3d09a
I20260619 02:49:16.012593 27622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: Bootstrap starting.
I20260619 02:49:16.013767 27622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:16.014443 27622 log.cc:824] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:16.015647 27622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: No bootstrap required, opened a new log
I20260619 02:49:16.018004 27622 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } }
I20260619 02:49:16.018198 27622 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:16.018234 27622 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8c74ad26ce48d1bcf1ca8e093f604d, State: Initialized, Role: FOLLOWER
I20260619 02:49:16.018612 27622 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } }
I20260619 02:49:16.018787 27622 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:16.018849 27622 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:16.018990 27622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:16.020347 27622 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } }
I20260619 02:49:16.020583 27622 leader_election.cc:302] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd8c74ad26ce48d1bcf1ca8e093f604d; no voters:
I20260619 02:49:16.020989 27622 leader_election.cc:288] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:16.021066 27627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:16.021291 27627 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [term 1 LEADER]: Becoming Leader. State: Replica: cd8c74ad26ce48d1bcf1ca8e093f604d, State: Running, Role: LEADER
I20260619 02:49:16.021377 27622 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:16.021426 27627 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } }
I20260619 02:49:16.022073 27629 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } } }
I20260619 02:49:16.022218 27629 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:16.022104 27627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd8c74ad26ce48d1bcf1ca8e093f604d. Latest consensus state: current_term: 1 leader_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8c74ad26ce48d1bcf1ca8e093f604d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35181 } } }
I20260619 02:49:16.022809 27627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:16.023458 27642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:16.023548 27642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:16.023602 27636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:16.023799 27636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:16.025475 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.011699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.26.98.62:35181)
0619 02:49:16.011957 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.011973 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.012020 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:49:16.012663 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.012675 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.012921 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.013178 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:16.014002 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.015173 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.016084 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.016595 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.016710 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.019961 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:16.019967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:16.019968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:16.019985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:16.023312 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.023949 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.023953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.023954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.024073 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.024349 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.024350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.024351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.024743 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:16.024924 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.025049 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.025228 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":122,"threads_started":1}
I20260619 02:49:16.025877 27636 catalog_manager.cc:1347] Generated new cluster ID: c8397a89da9d4811966a3e8e854872cf
I20260619 02:49:16.025956 27636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:16.035626 27636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:16.036522 27636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:16.041672 27636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd8c74ad26ce48d1bcf1ca8e093f604d: Generated new TSK 0
I20260619 02:49:16.041991 27636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:16.082311 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:16.216523 27648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:16.216866 27648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:16.216915 27648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:16.218576 27648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:16.218683 27648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:16.218732 27648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:16.221019 27648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:16.221393 27648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:16.221729 27648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:16.228056 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:16.228056 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:16.228193 27648 server_base.cc:1034] running on GCE node
W20260619 02:49:16.228391 27653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:16.228629 27648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:16.228950 27648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:16.230136 27648 hybrid_clock.cc:648] HybridClock initialized: now 1781837356230121 us; error 39 us; skew 500 ppm
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:16.233844 27648 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:16.234498 27648 webserver.cc:460] Webserver started at http://127.26.98.1:39453/ using document root <none> and password file <none>
I20260619 02:49:16.234766 27648 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:16.234835 27648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:16.235028 27648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:16.236353 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "ea7503004bef439da942663d8c20e4bc"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "5a46129c8aafb512ad11edb39d8dcd11"
server_key_iv: "0f51f3251c2d0196d08873ed47e62ecf"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.236754 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "ea7503004bef439da942663d8c20e4bc"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "5a46129c8aafb512ad11edb39d8dcd11"
server_key_iv: "0f51f3251c2d0196d08873ed47e62ecf"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.238615 27648 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:16.240125 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.240523 27648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:16.240640 27648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "ea7503004bef439da942663d8c20e4bc"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "5a46129c8aafb512ad11edb39d8dcd11"
server_key_iv: "0f51f3251c2d0196d08873ed47e62ecf"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.240746 27648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:16.287657 27648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:16.290855 27648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:16.291324 27648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:16.291689 27648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:16.292158 27648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:16.292207 27648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.292261 27648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:16.292289 27648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.300210 27648 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:43263
I20260619 02:49:16.300266 27776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:43263 every 8 connection(s)
I20260619 02:49:16.300655 27648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:16.301517 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27648
I20260619 02:49:16.301625 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:16.301826 27016 external_mini_cluster.cc:1428] Setting key 706c38b6a0859f38873bc799b7a7e73b
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:49:16.314266 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.301587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35071 (local address 127.26.98.62:35181)
0619 02:49:16.301675 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.301678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.302084 (+ 406us) server_negotiation.cc:407] Connection header received
0619 02:49:16.303333 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.303336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.303353 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.303446 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:16.304449 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.305027 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.306701 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.307043 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.307127 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.310388 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:16.310391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:16.310391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:16.310401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:16.312283 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.312903 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.312905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.312905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.313003 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.313349 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.313350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.313351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.313506 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:16.313574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.313899 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.314099 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:49:16.314399 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.301536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35181 (local address 127.26.98.1:35071)
0619 02:49:16.301830 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.301892 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.303197 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.303538 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.303544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.303876 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:16.304229 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.304243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.305177 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.305178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.306537 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.306542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.307159 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.307160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.307361 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.308066 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:16.308080 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:16.310195 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:16.312408 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.312418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.312421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.312780 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.313089 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.313090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.313090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.313268 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.313572 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:16.313581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:16.313758 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.313867 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.314104 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
I20260619 02:49:16.315047 27777 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35181
I20260619 02:49:16.315197 27777 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:16.315441 27777 heartbeater.cc:511] Master 127.26.98.62:35181 requested a full tablet report, sending...
I20260619 02:49:16.316195 27586 ts_manager.cc:194] Registered new tserver with Master: ea7503004bef439da942663d8c20e4bc (127.26.98.1:43263)
I20260619 02:49:16.316933 27586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:35071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:16.334717 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:16.437819 27782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:16.438342 27782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:16.438457 27782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:16.440671 27782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:16.440790 27782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:16.440848 27782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:16.443313 27782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:16.443686 27782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:16.444001 27782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:16.449779 27790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:16.449783 27787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:16.449941 27788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:16.450384 27782 server_base.cc:1034] running on GCE node
I20260619 02:49:16.450661 27782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:16.451056 27782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:16.452280 27782 hybrid_clock.cc:648] HybridClock initialized: now 1781837356452260 us; error 41 us; skew 500 ppm
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:16.456346 27782 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:16.456940 27782 webserver.cc:460] Webserver started at http://127.26.98.2:38079/ using document root <none> and password file <none>
I20260619 02:49:16.457373 27782 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:16.457475 27782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:16.457597 27782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:16.459084 27782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "d489bd1969184508a8d34b334e26f5fe"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "b097c1c60dd1ed05d3d3cf437c910d86"
server_key_iv: "77af957501d87a19a5020df227803c98"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.459443 27782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "d489bd1969184508a8d34b334e26f5fe"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "b097c1c60dd1ed05d3d3cf437c910d86"
server_key_iv: "77af957501d87a19a5020df227803c98"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.460992 27782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:49:16.462095 27797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.462450 27782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:16.462587 27782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "d489bd1969184508a8d34b334e26f5fe"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "b097c1c60dd1ed05d3d3cf437c910d86"
server_key_iv: "77af957501d87a19a5020df227803c98"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.462685 27782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:16.486357 27782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:16.489725 27782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:16.489899 27782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:16.490170 27782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:16.490586 27782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:16.490628 27782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.490664 27782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:16.490681 27782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.501121 27782 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:42657
I20260619 02:49:16.501166 27910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:42657 every 8 connection(s)
I20260619 02:49:16.501645 27782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:16.502197 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27782
I20260619 02:49:16.502297 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:16.502478 27016 external_mini_cluster.cc:1428] Setting key 9abdebec27fbc72ff9f9e56956bb27ac
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:49:16.515478 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.502630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:59733 (local address 127.26.98.62:35181)
0619 02:49:16.502782 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.502785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.503069 (+ 284us) server_negotiation.cc:407] Connection header received
0619 02:49:16.504681 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.504683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.504698 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.504756 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:16.505597 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.506156 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.507823 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.508234 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.508356 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.511607 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:16.511609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:16.511609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:16.511620 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:16.513600 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.514225 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.514227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.514227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.514288 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.514531 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.514532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.514533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.514663 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:16.514722 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.515273 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.515376 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 02:49:16.515712 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.502633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35181 (local address 127.26.98.2:59733)
0619 02:49:16.502859 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.502953 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.504506 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.504857 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.504863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.505216 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:16.505482 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.505488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.506263 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.506265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.507642 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.507647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.508401 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.508403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.508603 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.509384 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:16.509398 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:16.511448 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:16.513746 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.513763 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.513767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.514132 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.514345 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.514346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.514346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.514468 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.514724 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:16.514731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:16.514968 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.515188 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.515406 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
I20260619 02:49:16.516531 27911 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35181
I20260619 02:49:16.516696 27911 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:16.516953 27911 heartbeater.cc:511] Master 127.26.98.62:35181 requested a full tablet report, sending...
I20260619 02:49:16.517674 27586 ts_manager.cc:194] Registered new tserver with Master: d489bd1969184508a8d34b334e26f5fe (127.26.98.2:42657)
I20260619 02:49:16.518267 27586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:59733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:16.534474 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:16.632488 27916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:16.632795 27916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:16.632824 27916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:16.634552 27916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:16.634668 27916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:16.634711 27916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:16.637308 27916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:16.637678 27916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:16.637956 27916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:16.644212 27922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:16.644212 27924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:16.644212 27921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:16.644517 27916 server_base.cc:1034] running on GCE node
I20260619 02:49:16.644827 27916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:16.645130 27916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:16.646322 27916 hybrid_clock.cc:648] HybridClock initialized: now 1781837356646304 us; error 51 us; skew 500 ppm
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:16.650034 27916 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:16.650987 27916 webserver.cc:460] Webserver started at http://127.26.98.3:37675/ using document root <none> and password file <none>
I20260619 02:49:16.651319 27916 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:16.651562 27916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:16.652062 27916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:16.653578 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "f4bbf6c2b6844851baa77860597722ea"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "fc678047f4377184d96e83db254abd5c"
server_key_iv: "68d3d0d2f9551d9363d19676505a82da"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.653993 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "f4bbf6c2b6844851baa77860597722ea"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "fc678047f4377184d96e83db254abd5c"
server_key_iv: "68d3d0d2f9551d9363d19676505a82da"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.655925 27916 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260619 02:49:16.657096 27931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.657434 27916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:16.657544 27916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "f4bbf6c2b6844851baa77860597722ea"
format_stamp: "Formatted at 2026-06-19 02:49:16 on dist-test-slave-0f9d"
server_key: "fc678047f4377184d96e83db254abd5c"
server_key_iv: "68d3d0d2f9551d9363d19676505a82da"
server_key_version: "encryptionkey@0"
I20260619 02:49:16.657632 27916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:16.670074 27916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:16.673180 27916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:16.673331 27916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:16.673565 27916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:16.673873 27916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:16.673905 27916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.673936 27916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:16.673952 27916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:16.681717 27916 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:42307
I20260619 02:49:16.681793 28044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:42307 every 8 connection(s)
I20260619 02:49:16.682173 27916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:49:16 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:16.691519 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 27916
I20260619 02:49:16.691819 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:16.692078 27016 external_mini_cluster.cc:1428] Setting key d64daa6dde1d5baef344a9f10f609776
I20260619 02:49:16.695143 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.683146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:46423 (local address 127.26.98.62:35181)
0619 02:49:16.683212 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.683215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.683441 (+ 226us) server_negotiation.cc:407] Connection header received
0619 02:49:16.684602 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.684603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.684622 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.684678 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:16.685504 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.686067 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.687547 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.687803 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.687868 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.690853 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:16.690855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:16.690856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:16.690867 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:16.693004 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.693788 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.693790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.693790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.693872 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.694230 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.694231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.694231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.694377 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:16.694452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.694864 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.695037 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:16.695299 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.683120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35181 (local address 127.26.98.3:46423)
0619 02:49:16.683293 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.683338 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.684455 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.684799 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.684804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.685152 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:16.685405 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.685410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.686177 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.686178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.687424 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.687429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.687903 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.687904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.688061 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.688739 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:16.688751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:16.690649 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:16.693137 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.693160 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.693165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.693637 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.693975 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:16.693976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:16.693976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:16.694139 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:16.694462 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:16.694468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:16.694651 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.694784 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.695042 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260619 02:49:16.696203 28045 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35181
I20260619 02:49:16.696347 28045 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:16.696599 28045 heartbeater.cc:511] Master 127.26.98.62:35181 requested a full tablet report, sending...
I20260619 02:49:16.697228 27586 ts_manager.cc:194] Registered new tserver with Master: f4bbf6c2b6844851baa77860597722ea (127.26.98.3:42307)
I20260619 02:49:16.697789 27586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:46423
I20260619 02:49:16.705386 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:16.716280 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.707559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.26.98.62:35181)
0619 02:49:16.707658 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.707660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.707770 (+ 110us) server_negotiation.cc:407] Connection header received
0619 02:49:16.707850 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.707852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.707865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.707934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:16.708702 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.709267 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.710042 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.710249 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.710314 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.712255 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:16.712258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:16.712259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:16.712272 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:16.714097 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.714603 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.714605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.714605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.714681 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:16.715057 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:16.715060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:16.715060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:16.715334 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:16.715485 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.716045 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.716180 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:49:16.719902 27586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:16.722498 27586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:16.731221 28056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:42657 (local address 127.0.0.1:53216)
0619 02:49:16.726545 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.726604 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.726696 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.727114 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.727119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.727123 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:16.727240 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.727243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.728410 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.728412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.729871 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.729877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.730773 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.730774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.730937 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.730960 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.731038 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.731119 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":77,"thread_start_us":142,"threads_started":1}
I20260619 02:49:16.731408 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:43263 (local address 127.0.0.1:56234)
0619 02:49:16.726543 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.726604 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.726696 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.727194 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.727195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.727197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:16.727246 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.727247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.728649 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.728650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.729915 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.729917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.731110 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.731112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.731184 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.731188 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.731258 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.731314 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":7,"thread_start_us":159,"threads_started":1}
I20260619 02:49:16.731787 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.26.98.2:42657)
0619 02:49:16.726851 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.726856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.726915 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:49:16.726969 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.726972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.726991 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.727321 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:16.727400 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.728243 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.729993 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.730676 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.730758 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.730843 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.731609 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.731680 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":110,"threads_started":1}
I20260619 02:49:16.732061 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.26.98.1:43263)
0619 02:49:16.726941 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.726944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.726954 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:16.726994 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.726996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.727012 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.727096 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:16.727331 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.728544 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.730046 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.730933 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.731003 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.731069 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.731797 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.731971 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":92,"threads_started":1}
I20260619 02:49:16.732210 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:42307 (local address 127.0.0.1:47490)
0619 02:49:16.726807 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.726815 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.726863 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.727211 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.727212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.727214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:16.727265 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.727266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.729636 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.729637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.730717 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.730720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.731928 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.731929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.732011 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.732016 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.732064 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.732116 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260619 02:49:16.732506 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.726508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.26.98.3:42307)
0619 02:49:16.726746 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.726752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.726935 (+ 183us) server_negotiation.cc:407] Connection header received
0619 02:49:16.727024 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.727026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.727044 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.727114 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:16.727337 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.728528 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.730845 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.731732 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.731804 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.731872 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.732377 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.732431 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
I20260619 02:49:16.732939 27845 tablet_service.cc:1467] Processing CreateTablet for tablet b6d57cafb66a448181526d545d7327b2 (DEFAULT_TABLE table=test-table [id=e94ecf5bca7a41c4b2ae45d5a60f82e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:16.733037 27711 tablet_service.cc:1467] Processing CreateTablet for tablet b6d57cafb66a448181526d545d7327b2 (DEFAULT_TABLE table=test-table [id=e94ecf5bca7a41c4b2ae45d5a60f82e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:16.733208 27979 tablet_service.cc:1467] Processing CreateTablet for tablet b6d57cafb66a448181526d545d7327b2 (DEFAULT_TABLE table=test-table [id=e94ecf5bca7a41c4b2ae45d5a60f82e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:16.733412 27711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d57cafb66a448181526d545d7327b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:16.733413 27845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d57cafb66a448181526d545d7327b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:16.733547 27979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d57cafb66a448181526d545d7327b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:16.736821 28062 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Bootstrap starting.
I20260619 02:49:16.737501 28064 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Bootstrap starting.
I20260619 02:49:16.737769 28062 tablet_bootstrap.cc:654] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:16.738293 28064 tablet_bootstrap.cc:654] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:16.738588 28062 log.cc:824] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:16.738963 28064 log.cc:824] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:16.739104 28063 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Bootstrap starting.
I20260619 02:49:16.739784 28063 tablet_bootstrap.cc:654] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:16.740227 28062 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: No bootstrap required, opened a new log
I20260619 02:49:16.740324 28062 ts_tablet_manager.cc:1397] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260619 02:49:16.740312 28064 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: No bootstrap required, opened a new log
I20260619 02:49:16.740417 28064 ts_tablet_manager.cc:1397] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:49:16.740518 28063 log.cc:824] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:16.741680 28063 tablet_bootstrap.cc:492] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: No bootstrap required, opened a new log
I20260619 02:49:16.741752 28063 ts_tablet_manager.cc:1397] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:16.742480 28064 raft_consensus.cc:348] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.742678 28064 raft_consensus.cc:374] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:16.742702 28064 raft_consensus.cc:729] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bbf6c2b6844851baa77860597722ea, State: Initialized, Role: FOLLOWER
I20260619 02:49:16.742830 28064 consensus_queue.cc:260] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.742769 28062 raft_consensus.cc:348] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.743222 28064 ts_tablet_manager.cc:1428] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 02:49:16.743253 28062 raft_consensus.cc:374] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:16.743299 28062 raft_consensus.cc:729] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7503004bef439da942663d8c20e4bc, State: Initialized, Role: FOLLOWER
I20260619 02:49:16.743355 28045 heartbeater.cc:503] Master 127.26.98.62:35181 was elected leader, sending a full tablet report...
I20260619 02:49:16.743532 28062 consensus_queue.cc:260] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.743841 28062 ts_tablet_manager.cc:1428] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:16.743918 27777 heartbeater.cc:503] Master 127.26.98.62:35181 was elected leader, sending a full tablet report...
I20260619 02:49:16.744415 28063 raft_consensus.cc:348] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.744630 28063 raft_consensus.cc:374] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:16.744678 28063 raft_consensus.cc:729] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d489bd1969184508a8d34b334e26f5fe, State: Initialized, Role: FOLLOWER
I20260619 02:49:16.744805 28063 consensus_queue.cc:260] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.745129 28063 ts_tablet_manager.cc:1428] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:16.745126 27911 heartbeater.cc:503] Master 127.26.98.62:35181 was elected leader, sending a full tablet report...
W20260619 02:49:16.752431 27912 tablet.cc:2369] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:16.801604 27778 tablet.cc:2369] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:16.932950 28046 tablet.cc:2369] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:16.983104 28070 raft_consensus.cc:482] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:16.983280 28070 raft_consensus.cc:504] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.983741 28070 leader_election.cc:288] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4bbf6c2b6844851baa77860597722ea (127.26.98.3:42307), ea7503004bef439da942663d8c20e4bc (127.26.98.1:43263)
I20260619 02:49:16.987111 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.984231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:50249 (local address 127.26.98.1:43263)
0619 02:49:16.984298 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.984301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.984318 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:16.984358 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.984359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.984373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.984431 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:16.984625 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.985165 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.986238 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.986634 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.986706 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.986721 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.986948 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.987008 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:49:16.987315 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.984250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:60741 (local address 127.26.98.3:42307)
0619 02:49:16.984304 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:16.984306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:16.984322 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:49:16.984479 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:16.984481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:16.984496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:16.984578 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:16.984784 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.985279 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.986281 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.986769 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.986972 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.986994 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:16.987183 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:49:16.987248 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:16.987473 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.984034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:43263 (local address 127.26.98.2:50249)
0619 02:49:16.984159 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.984177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.984294 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.984476 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.984478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.984480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:16.984562 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.984564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.985249 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.985251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.986126 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.986130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.986714 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.986715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.986803 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.986808 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.987318 (+ 510us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.987382 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":3}
I20260619 02:49:16.987556 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:16.984034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:42307 (local address 127.26.98.2:60741)
0619 02:49:16.984252 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:16.984265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:16.984355 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:16.984600 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:16.984601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:16.984604 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:16.984713 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.984716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.985365 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.985366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.986189 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:16.986191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:16.987003 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:16.987004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:16.987085 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:16.987089 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:16.987413 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 02:49:16.987463 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":30,"thread_start_us":79,"threads_started":1}
I20260619 02:49:16.987955 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d57cafb66a448181526d545d7327b2" candidate_uuid: "d489bd1969184508a8d34b334e26f5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bbf6c2b6844851baa77860597722ea" is_pre_election: true
I20260619 02:49:16.987975 27731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d57cafb66a448181526d545d7327b2" candidate_uuid: "d489bd1969184508a8d34b334e26f5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea7503004bef439da942663d8c20e4bc" is_pre_election: true
I20260619 02:49:16.988183 27998 raft_consensus.cc:2454] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d489bd1969184508a8d34b334e26f5fe in term 0.
I20260619 02:49:16.988201 27731 raft_consensus.cc:2454] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d489bd1969184508a8d34b334e26f5fe in term 0.
I20260619 02:49:16.988478 27801 leader_election.cc:302] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d489bd1969184508a8d34b334e26f5fe, f4bbf6c2b6844851baa77860597722ea; no voters:
I20260619 02:49:16.988662 28070 raft_consensus.cc:2781] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:16.988715 28070 raft_consensus.cc:482] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:16.988731 28070 raft_consensus.cc:3037] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:16.989615 28070 raft_consensus.cc:504] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.989771 28070 leader_election.cc:288] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [CANDIDATE]: Term 1 election: Requested vote from peers f4bbf6c2b6844851baa77860597722ea (127.26.98.3:42307), ea7503004bef439da942663d8c20e4bc (127.26.98.1:43263)
I20260619 02:49:16.990195 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d57cafb66a448181526d545d7327b2" candidate_uuid: "d489bd1969184508a8d34b334e26f5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bbf6c2b6844851baa77860597722ea"
I20260619 02:49:16.990180 27731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d57cafb66a448181526d545d7327b2" candidate_uuid: "d489bd1969184508a8d34b334e26f5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea7503004bef439da942663d8c20e4bc"
I20260619 02:49:16.990314 27998 raft_consensus.cc:3037] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:16.990316 27731 raft_consensus.cc:3037] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:16.991636 27998 raft_consensus.cc:2454] T b6d57cafb66a448181526d545d7327b2 P f4bbf6c2b6844851baa77860597722ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d489bd1969184508a8d34b334e26f5fe in term 1.
I20260619 02:49:16.991636 27731 raft_consensus.cc:2454] T b6d57cafb66a448181526d545d7327b2 P ea7503004bef439da942663d8c20e4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d489bd1969184508a8d34b334e26f5fe in term 1.
I20260619 02:49:16.991961 27801 leader_election.cc:302] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d489bd1969184508a8d34b334e26f5fe, f4bbf6c2b6844851baa77860597722ea; no voters:
I20260619 02:49:16.992106 28070 raft_consensus.cc:2781] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:16.992331 28070 raft_consensus.cc:686] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [term 1 LEADER]: Becoming Leader. State: Replica: d489bd1969184508a8d34b334e26f5fe, State: Running, Role: LEADER
I20260619 02:49:16.992514 28070 consensus_queue.cc:237] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } }
I20260619 02:49:16.993834 27585 catalog_manager.cc:5495] T b6d57cafb66a448181526d545d7327b2 P d489bd1969184508a8d34b334e26f5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to d489bd1969184508a8d34b334e26f5fe (127.26.98.2). New cstate: current_term: 1 leader_uuid: "d489bd1969184508a8d34b334e26f5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4bbf6c2b6844851baa77860597722ea" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d489bd1969184508a8d34b334e26f5fe" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea7503004bef439da942663d8c20e4bc" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43263 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:17.026945 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.015853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.26.98.1:43263)
0619 02:49:17.015936 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:17.015938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:17.016153 (+ 215us) server_negotiation.cc:407] Connection header received
0619 02:49:17.016190 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:17.016191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:17.016204 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:17.016255 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:17.017196 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.017719 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.018609 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.018827 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.019067 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.022600 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:17.022617 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:17.022617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:17.022630 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:17.025016 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.025580 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.025585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.025585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.025670 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.026121 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.026122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.026123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.026347 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:17.026494 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:17.026692 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:49:17.026787 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:17.057171 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.047528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.26.98.1:43263)
0619 02:49:17.047658 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:17.047660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:17.047860 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:49:17.047910 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:17.047911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:17.047931 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:17.048000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:17.048778 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.049309 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.050308 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.050541 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.050611 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.054001 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:17.054003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:17.054004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:17.054014 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:17.055698 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.056163 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.056165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.056165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.056232 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.056459 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.056459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.056460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.056593 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:17.056722 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:17.056952 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 02:49:17.057059 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:17.085677 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.077027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.26.98.1:43263)
0619 02:49:17.077119 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:17.077121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:17.077256 (+ 135us) server_negotiation.cc:407] Connection header received
0619 02:49:17.077355 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:17.077356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:17.077367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:17.077411 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:17.078043 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.078565 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.079495 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.079706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.079774 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.082641 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:17.082644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:17.082644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:17.082656 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:17.084384 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.084849 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.084850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.084851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.084928 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.085109 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.085110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.085111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.085237 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:17.085335 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:17.085457 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:49:17.085560 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:49:17.086412 27691 tablet_service.cc:2908] Rejecting scan request for tablet b6d57cafb66a448181526d545d7327b2: Uninitialized: clean time has not yet been initialized
I20260619 02:49:17.087697 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27648
I20260619 02:49:17.094046 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27782
I20260619 02:49:17.100131 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27916
I20260619 02:49:17.105434 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 27554
2026-06-19T02:49:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2390 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[28086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[28086](info): set up 2 sockets
Jun 19 02:49:17 dist-test-slave-0f9d krb5kdc[28086](info): commencing operation
krb5kdc: starting...
W20260619 02:49:18.206434 27016 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 02:49:18.541656 27016 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.335s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 02:49:18.841858 27016 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.300s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 02:49:19.117208 27016 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.275s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/test-user.keytab.
W20260619 02:49:19.327016 27016 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.210s user 0.000s sys 0.002s
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:19.382637 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35193
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:19.478695 28099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:19.479043 28099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:19.479081 28099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:19.481201 28099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:19.481315 28099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:49:19.481336 28099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:19.481353 28099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:19.481369 28099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:19.481383 28099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:19.483899 28099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35193
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:19.484356 28099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:19.484658 28099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:19.490656 28107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.490656 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.490681 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:19.491007 28099 server_base.cc:1034] running on GCE node
I20260619 02:49:19.491274 28099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:19.491632 28099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:19.492868 28099 hybrid_clock.cc:648] HybridClock initialized: now 1781837359492845 us; error 46 us; skew 500 ppm
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:19.500049 28099 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:19.500643 28099 webserver.cc:460] Webserver started at http://127.26.98.62:35431/ using document root <none> and password file <none>
I20260619 02:49:19.500932 28099 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:19.500988 28099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:19.501144 28099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:19.502482 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "1eaf68fbce214bfebcf6aaa731cf81b5"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "b0fb06672e28ca20e779354ec406c5f9"
server_key_iv: "e6b0f358198b0b4a38092fe13f93d277"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.502923 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "1eaf68fbce214bfebcf6aaa731cf81b5"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "b0fb06672e28ca20e779354ec406c5f9"
server_key_iv: "e6b0f358198b0b4a38092fe13f93d277"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.504541 28099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:19.505719 28114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:19.505944 28099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:19.506042 28099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "1eaf68fbce214bfebcf6aaa731cf81b5"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "b0fb06672e28ca20e779354ec406c5f9"
server_key_iv: "e6b0f358198b0b4a38092fe13f93d277"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.506137 28099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:19.528633 28099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:19.532770 28099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:19.532965 28099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:19.538936 28099 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35193
I20260619 02:49:19.539003 28176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35193 every 8 connection(s)
I20260619 02:49:19.539449 28099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:19.539819 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28099
I20260619 02:49:19.539937 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:19.540158 27016 external_mini_cluster.cc:1428] Setting key 9ad12c4d0402e00acd531f64ee2cefd3
I20260619 02:49:19.540390 28177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:19.544508 28177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Bootstrap starting.
I20260619 02:49:19.545521 28177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:19.545838 28177 log.cc:824] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:19.546721 28177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: No bootstrap required, opened a new log
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:19.549801 28177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } }
I20260619 02:49:19.550047 28177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:19.550087 28177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaf68fbce214bfebcf6aaa731cf81b5, State: Initialized, Role: FOLLOWER
I20260619 02:49:19.550254 28177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } }
I20260619 02:49:19.550365 28177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:19.550405 28177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:19.550460 28177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:19.551386 28177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } }
I20260619 02:49:19.551517 28177 leader_election.cc:302] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eaf68fbce214bfebcf6aaa731cf81b5; no voters:
I20260619 02:49:19.551748 28177 leader_election.cc:288] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:19.551803 28182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:19.551935 28182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [term 1 LEADER]: Becoming Leader. State: Replica: 1eaf68fbce214bfebcf6aaa731cf81b5, State: Running, Role: LEADER
I20260619 02:49:19.552024 28182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } }
I20260619 02:49:19.552093 28177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:19.552392 28182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eaf68fbce214bfebcf6aaa731cf81b5. Latest consensus state: current_term: 1 leader_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } } }
I20260619 02:49:19.552475 28182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:19.553357 28187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:19.553615 28183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaf68fbce214bfebcf6aaa731cf81b5" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35193 } } }
I20260619 02:49:19.554029 28183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:19.554047 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.540778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.26.98.62:35193)
0619 02:49:19.540980 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.540991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.541031 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:19.541780 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.541793 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.541884 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.542051 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.543070 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.544168 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.545393 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.546085 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.546234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.549587 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.549595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.549595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.549613 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.551733 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.552230 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.552234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.552235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.552370 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.552600 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.552601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.552602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.552983 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.553168 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.553625 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.553806 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260619 02:49:19.554147 28187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:19.556059 28187 catalog_manager.cc:1347] Generated new cluster ID: 4b11709627044036a8be51afa436ea52
I20260619 02:49:19.556152 28187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:19.561045 28187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:19.561709 28187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:19.568112 28187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Generated new TSK 0
I20260619 02:49:19.568377 28187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:19.607163 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35193
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:19.727557 28203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:19.727957 28203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:19.728006 28203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:19.730661 28203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:19.730767 28203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:19.730798 28203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:19.730820 28203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:19.733184 28203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:19.733559 28203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:19.733839 28203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:19.740224 28211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.740373 28209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.740224 28208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:19.740775 28203 server_base.cc:1034] running on GCE node
I20260619 02:49:19.741050 28203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:19.741387 28203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:19.742575 28203 hybrid_clock.cc:648] HybridClock initialized: now 1781837359742558 us; error 36 us; skew 500 ppm
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:19.746174 28203 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:19.746826 28203 webserver.cc:460] Webserver started at http://127.26.98.1:36921/ using document root <none> and password file <none>
I20260619 02:49:19.747133 28203 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:19.747195 28203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:19.747349 28203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:19.748698 28203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "354d7b7156b74c7f93ad4a11938822b4"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "8f9c88689b0ac1747c374899138e5c79"
server_key_iv: "99309c0a6006d67298521b3ae51f24bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.749137 28203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "354d7b7156b74c7f93ad4a11938822b4"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "8f9c88689b0ac1747c374899138e5c79"
server_key_iv: "99309c0a6006d67298521b3ae51f24bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.750774 28203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 02:49:19.751909 28218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:19.752255 28203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:19.752357 28203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "354d7b7156b74c7f93ad4a11938822b4"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "8f9c88689b0ac1747c374899138e5c79"
server_key_iv: "99309c0a6006d67298521b3ae51f24bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.752509 28203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:19.783742 28203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:19.786479 28203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:19.786640 28203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:19.787338 28203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:19.787396 28203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:19.787431 28203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:19.787448 28203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:19.795975 28203 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:35885
I20260619 02:49:19.796142 28333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:35885 every 8 connection(s)
I20260619 02:49:19.796471 28203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:19.799675 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.787954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:48159 (local address 127.26.98.62:35193)
0619 02:49:19.788020 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.788022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.788343 (+ 321us) server_negotiation.cc:407] Connection header received
0619 02:49:19.789034 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.789036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.789056 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.789140 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.790308 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.791241 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.792740 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.793129 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.793225 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.796326 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.796328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.796329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.796340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.797934 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.798538 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.798539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.798539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.798611 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.798936 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.798937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.798938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.799085 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.799154 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.799493 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.799592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:49:19.799839 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.787966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.1:48159)
0619 02:49:19.788183 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.788228 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.788864 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.789245 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.789251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.789770 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:19.790154 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.790163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.791405 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.791406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.792613 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.792618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.793265 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.793266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.793416 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.794144 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:19.794156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:19.796184 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:19.798111 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.798119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.798122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.798431 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.798679 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.798680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.798680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.798803 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.799148 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:19.799153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:19.799335 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 02:49:19.799437 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:49:19.799604 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260619 02:49:19.800390 28334 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35193
I20260619 02:49:19.800483 28334 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:19.800710 28334 heartbeater.cc:511] Master 127.26.98.62:35193 requested a full tablet report, sending...
I20260619 02:49:19.801355 28130 ts_manager.cc:194] Registered new tserver with Master: 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885)
I20260619 02:49:19.802250 28130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:48159
I20260619 02:49:19.805869 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28203
I20260619 02:49:19.805995 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:19.806186 27016 external_mini_cluster.cc:1428] Setting key a5b6a242b120eb5e561d62b339a47653
I20260619 02:49:19.808763 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.801741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:38189 (local address 127.26.98.62:35193)
0619 02:49:19.801795 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.801797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.801930 (+ 133us) server_negotiation.cc:407] Connection header received
0619 02:49:19.801977 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.801978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.801990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.802043 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.802408 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.803063 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.803850 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.804078 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.804147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.805590 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.805592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.805593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.805604 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.807565 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.807944 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.807945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.807946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.808012 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.808188 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.808189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.808189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.808328 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.808391 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.808533 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.808639 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:49:19.808763 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.801637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.1:38189)
0619 02:49:19.801849 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.801860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.801927 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.802048 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.802049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.802256 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:19.802329 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.802332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.803158 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.803160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.803748 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.803752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.804176 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.804177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.804266 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.804990 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:19.804996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:19.805379 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:19.807662 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.807664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.807664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.807871 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.808072 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.808072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.808073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.808134 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.808386 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:19.808387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:19.808444 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:49:19.808503 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:49:19.808643 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:19.837224 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35193
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:19.934448 28343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:19.934773 28343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:19.934818 28343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:19.936471 28343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:19.936548 28343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:19.936586 28343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:19.936614 28343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:19.938601 28343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:19.939007 28343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:19.939280 28343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:19.945250 28348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.945250 28349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:19.945250 28351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:19.945557 28343 server_base.cc:1034] running on GCE node
I20260619 02:49:19.945971 28343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:19.946439 28343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:19.947619 28343 hybrid_clock.cc:648] HybridClock initialized: now 1781837359947566 us; error 85 us; skew 500 ppm
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:19.951184 28343 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:19.951818 28343 webserver.cc:460] Webserver started at http://127.26.98.2:35149/ using document root <none> and password file <none>
I20260619 02:49:19.952069 28343 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:19.952126 28343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:19.952272 28343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:19.953660 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "83bd29c7dfdc455caf31b9ce20f45127"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "00191a6f192bc2344e014fcc21c3ef4c"
server_key_iv: "5ca37a38850a6b42012ea17c1ff58390"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.954085 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "83bd29c7dfdc455caf31b9ce20f45127"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "00191a6f192bc2344e014fcc21c3ef4c"
server_key_iv: "5ca37a38850a6b42012ea17c1ff58390"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.955790 28343 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 02:49:19.956880 28358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:19.957288 28343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:19.957388 28343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "83bd29c7dfdc455caf31b9ce20f45127"
format_stamp: "Formatted at 2026-06-19 02:49:19 on dist-test-slave-0f9d"
server_key: "00191a6f192bc2344e014fcc21c3ef4c"
server_key_iv: "5ca37a38850a6b42012ea17c1ff58390"
server_key_version: "encryptionkey@0"
I20260619 02:49:19.957495 28343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:19.974745 28343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:19.977854 28343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:19.978016 28343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:19.978677 28343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:19.978720 28343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:19.978746 28343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:19.978758 28343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:19 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:19.987665 28343 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:39307
I20260619 02:49:19.987788 28473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:39307 every 8 connection(s)
I20260619 02:49:19.988232 28343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:19.991480 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.979213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:32865 (local address 127.26.98.62:35193)
0619 02:49:19.979296 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.979299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.979530 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:49:19.980150 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.980152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.980168 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.980275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.981097 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.981914 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.983288 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.983620 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.983729 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.987281 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.987284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.987284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.987297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.989110 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.989846 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.989849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.989849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.989975 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.990565 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.990567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.990567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.990739 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.990811 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.991273 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.991398 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:19.991741 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.979182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.2:32865)
0619 02:49:19.979372 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.979422 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.980038 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.980349 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.980355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.980667 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:19.980985 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.980993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.982059 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.982060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.983175 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.983180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.983787 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.983789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.983986 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.984735 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:19.984761 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:19.987114 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:19.989317 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.989330 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.989333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.989671 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.990143 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.990145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.990145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.990362 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:19.990834 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:19.990841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:19.991075 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 02:49:19.991200 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:49:19.991442 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260619 02:49:19.992604 28474 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35193
I20260619 02:49:19.992777 28474 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:19.993242 28474 heartbeater.cc:511] Master 127.26.98.62:35193 requested a full tablet report, sending...
I20260619 02:49:19.993765 28130 ts_manager.cc:194] Registered new tserver with Master: 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307)
I20260619 02:49:19.994127 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28343
I20260619 02:49:19.994231 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:19.994318 28130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:32865
I20260619 02:49:19.994428 27016 external_mini_cluster.cc:1428] Setting key 2a3330453301e81e642b65e60be9c566
I20260619 02:49:20.000895 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.993891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.2:38193)
0619 02:49:19.994110 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.994122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.994246 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.994399 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.994400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.994656 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:19.994753 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.994757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.995666 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.995667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.996416 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.996420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.996865 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.996866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.996957 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.997547 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:19.997551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:19.997935 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:19.999686 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:19.999687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:19.999688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:19.999894 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.000082 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.000083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.000083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.000151 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.000515 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:20.000517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:20.000579 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.000641 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.000777 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260619 02:49:20.000921 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.993977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:38193 (local address 127.26.98.62:35193)
0619 02:49:19.994028 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.994030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.994201 (+ 171us) server_negotiation.cc:407] Connection header received
0619 02:49:19.994317 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.994318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.994331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.994394 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.994842 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.995541 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.996526 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.996755 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.996844 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.998049 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.998051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.998051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.998059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.999566 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.999965 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.999966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.999966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.000024 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.000212 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.000212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.000213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.000428 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.000493 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.000679 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.000774 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:20.025914 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35193
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:20.130304 28483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:20.130661 28483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:20.130699 28483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:20.134539 28483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:20.134694 28483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:20.136536 28483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:20.136632 28483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:20.140139 28483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:20.140523 28483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:20.140802 28483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:20.147322 28489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.147322 28488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.148814 28491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:20.148998 28483 server_base.cc:1034] running on GCE node
I20260619 02:49:20.149549 28483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:20.150053 28483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:20.151247 28483 hybrid_clock.cc:648] HybridClock initialized: now 1781837360151208 us; error 74 us; skew 500 ppm
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837360, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:20.154744 28483 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:20.155413 28483 webserver.cc:460] Webserver started at http://127.26.98.3:46085/ using document root <none> and password file <none>
I20260619 02:49:20.155769 28483 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:20.155826 28483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:20.155974 28483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:20.157315 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "cd2283255d7f4888ab3b538b76fd6fbb"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-0f9d"
server_key: "0b93cbdebf48f4df53226ce4a333c475"
server_key_iv: "3d2e40388a73b62615ac31a3167553f4"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.157698 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "cd2283255d7f4888ab3b538b76fd6fbb"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-0f9d"
server_key: "0b93cbdebf48f4df53226ce4a333c475"
server_key_iv: "3d2e40388a73b62615ac31a3167553f4"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.159396 28483 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:49:20.160413 28498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.160633 28483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.160728 28483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "cd2283255d7f4888ab3b538b76fd6fbb"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-0f9d"
server_key: "0b93cbdebf48f4df53226ce4a333c475"
server_key_iv: "3d2e40388a73b62615ac31a3167553f4"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.160804 28483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:20.187742 28483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:20.191041 28483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:20.191202 28483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:20.191872 28483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:20.191924 28483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.191951 28483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:20.191962 28483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.200709 28483 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:46175
I20260619 02:49:20.200829 28613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:46175 every 8 connection(s)
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837360, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:20.201252 28483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:20.204005 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28483
I20260619 02:49:20.204119 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:20.204336 27016 external_mini_cluster.cc:1428] Setting key 21b9e1f49562def5790846ce8919ee5f
I20260619 02:49:20.205993 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.192971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:33349 (local address 127.26.98.62:35193)
0619 02:49:20.193062 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.193066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.193325 (+ 259us) server_negotiation.cc:407] Connection header received
0619 02:49:20.194314 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.194316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.194336 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.194447 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.195641 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.196510 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.197926 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.198270 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.198401 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.202072 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.202074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.202074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.202085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.203645 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.204310 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.204312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.204312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.204416 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.205012 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.205014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.205015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.205250 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.205353 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.205773 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.205892 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:20.206249 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.192924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.3:33349)
0619 02:49:20.193171 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.193217 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.194089 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.194567 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.194575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.195088 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:20.195474 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.195482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.196645 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.196646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.197811 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.197815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.198427 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.198428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.198622 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.199428 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:20.199441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:20.201933 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:20.203773 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.203786 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.203790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.204193 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.204530 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.204532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.204533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.204851 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.205358 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:20.205367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:20.205585 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.205724 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.205954 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":144,"threads_started":1}
I20260619 02:49:20.206935 28614 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35193
I20260619 02:49:20.207063 28614 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:20.207312 28614 heartbeater.cc:511] Master 127.26.98.62:35193 requested a full tablet report, sending...
I20260619 02:49:20.207839 28131 ts_manager.cc:194] Registered new tserver with Master: cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175)
I20260619 02:49:20.208312 28131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:33349
I20260619 02:49:20.216856 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.208422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:40217 (local address 127.26.98.62:35193)
0619 02:49:20.208489 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.208492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.208599 (+ 107us) server_negotiation.cc:407] Connection header received
0619 02:49:20.208680 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.208681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.208690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.208748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.209135 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.209748 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.210859 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.211129 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.211206 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.213266 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.213270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.213270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.213284 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.215188 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.215733 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.215734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.215735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.215866 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.216158 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.216160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.216160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.216356 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.216444 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.216630 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.216743 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:49:20.216879 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.208336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.26.98.3:40217)
0619 02:49:20.208535 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.208549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.208616 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.208753 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.208754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.208967 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:20.209052 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.209054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.210024 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.210025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.210628 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.210632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.211227 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.211228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.211307 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.212645 (+ 1338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:20.212652 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:20.213090 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:20.215346 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.215348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.215349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.215607 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.215958 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.215959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.215960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.216053 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.216448 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:20.216449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:20.216513 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.216588 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.216749 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
I20260619 02:49:20.216943 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:20.244658 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.235940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.26.98.62:35193)
0619 02:49:20.236010 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.236012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.236019 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:20.236048 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.236049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.236059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.236109 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.236763 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.237301 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.238063 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.238263 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.238324 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.241133 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.241135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.241136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.241146 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.242864 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.243476 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.243479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.243479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.243576 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.243827 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.243828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.243829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.243997 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.244119 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.244437 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.244539 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:49:20.247395 28131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:20.249483 28131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:20.257718 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:35885 (local address 127.0.0.1:34460)
0619 02:49:20.252706 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.252719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.252787 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.253292 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.253298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.253300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.253412 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.253415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.254729 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.254730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.256070 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.256072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257331 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.257443 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.257447 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.257504 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.257592 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":45,"thread_start_us":118,"threads_started":1}
I20260619 02:49:20.257727 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.26.98.3:46175)
0619 02:49:20.253100 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.253104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.253114 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:20.253159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.253161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.253179 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.253268 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.253579 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.254433 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.256187 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257193 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.257374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.257567 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.257622 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260619 02:49:20.257848 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.26.98.2:39307)
0619 02:49:20.253030 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.253036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.253542 (+ 506us) server_negotiation.cc:407] Connection header received
0619 02:49:20.253591 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.253593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.253610 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.253686 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.253922 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.254987 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.256137 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257190 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.257629 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.257658 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.257722 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":145,"threads_started":1}
I20260619 02:49:20.258212 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.26.98.1:35885)
0619 02:49:20.253113 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.253117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.253128 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:20.253176 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.253178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.253194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.253342 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.253484 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.254642 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.256188 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257219 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257900 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.258009 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.258039 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.258095 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":60,"threads_started":1}
I20260619 02:49:20.258572 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:46175 (local address 127.0.0.1:56672)
0619 02:49:20.252608 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.252655 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.252738 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.253273 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.253305 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.253307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.253509 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.253510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.254512 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.254513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.256069 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.256072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257407 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.257470 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.257473 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.257519 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.258486 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":781,"thread_start_us":71,"threads_started":1}
I20260619 02:49:20.258738 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.252659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:39307 (local address 127.0.0.1:38564)
0619 02:49:20.253441 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.253451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.253508 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.253753 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.253753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.253755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.253828 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.253829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.255119 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.255120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.256052 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.256055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.257322 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.257324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.257427 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.257446 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.257514 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.258663 (+ 1149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1075,"thread_start_us":85,"threads_started":1}
I20260619 02:49:20.259264 28268 tablet_service.cc:1467] Processing CreateTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 (DEFAULT_TABLE table=test-table [id=dcad63afca684eaea1fd9d8be53b744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.259581 28548 tablet_service.cc:1467] Processing CreateTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 (DEFAULT_TABLE table=test-table [id=dcad63afca684eaea1fd9d8be53b744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.259826 28268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334a0e442db24f7a9bb9888487cb70e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.259945 28548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334a0e442db24f7a9bb9888487cb70e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.259619 28408 tablet_service.cc:1467] Processing CreateTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 (DEFAULT_TABLE table=test-table [id=dcad63afca684eaea1fd9d8be53b744c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.260211 28408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334a0e442db24f7a9bb9888487cb70e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.264125 28639 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Bootstrap starting.
I20260619 02:49:20.264505 28638 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Bootstrap starting.
I20260619 02:49:20.264617 28637 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Bootstrap starting.
I20260619 02:49:20.265163 28639 tablet_bootstrap.cc:654] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.265506 28639 log.cc:824] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:20.265779 28638 tablet_bootstrap.cc:654] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.265884 28637 tablet_bootstrap.cc:654] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.266170 28638 log.cc:824] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:20.266242 28637 log.cc:824] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:20.266400 28639 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: No bootstrap required, opened a new log
I20260619 02:49:20.266475 28639 ts_tablet_manager.cc:1397] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:20.267153 28638 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: No bootstrap required, opened a new log
I20260619 02:49:20.267304 28638 ts_tablet_manager.cc:1397] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:20.267333 28637 tablet_bootstrap.cc:492] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: No bootstrap required, opened a new log
I20260619 02:49:20.267418 28637 ts_tablet_manager.cc:1397] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:20.269006 28639 raft_consensus.cc:348] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.269209 28639 raft_consensus.cc:374] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.269246 28639 raft_consensus.cc:729] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bd29c7dfdc455caf31b9ce20f45127, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.269425 28639 consensus_queue.cc:260] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.269829 28474 heartbeater.cc:503] Master 127.26.98.62:35193 was elected leader, sending a full tablet report...
I20260619 02:49:20.269851 28639 ts_tablet_manager.cc:1428] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:20.270546 28638 raft_consensus.cc:348] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.270783 28638 raft_consensus.cc:374] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.270711 28637 raft_consensus.cc:348] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.270833 28638 raft_consensus.cc:729] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354d7b7156b74c7f93ad4a11938822b4, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.270937 28637 raft_consensus.cc:374] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.270982 28637 raft_consensus.cc:729] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2283255d7f4888ab3b538b76fd6fbb, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.271070 28638 consensus_queue.cc:260] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.271292 28637 consensus_queue.cc:260] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.271469 28638 ts_tablet_manager.cc:1428] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:20.271570 28334 heartbeater.cc:503] Master 127.26.98.62:35193 was elected leader, sending a full tablet report...
I20260619 02:49:20.272181 28614 heartbeater.cc:503] Master 127.26.98.62:35193 was elected leader, sending a full tablet report...
I20260619 02:49:20.272305 28637 ts_tablet_manager.cc:1428] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
W20260619 02:49:20.297433 28335 tablet.cc:2369] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:20.452153 28615 tablet.cc:2369] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:20.489344 28475 tablet.cc:2369] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:20.501266 28645 raft_consensus.cc:482] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:20.501451 28645 raft_consensus.cc:504] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.501906 28645 leader_election.cc:288] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885), 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307)
I20260619 02:49:20.505070 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.502222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:34429 (local address 127.26.98.1:35885)
0619 02:49:20.502276 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.502278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.502428 (+ 150us) server_negotiation.cc:407] Connection header received
0619 02:49:20.502484 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.502485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.502498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.502548 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.502695 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.503201 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504276 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.504686 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504754 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.504769 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.504929 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.504975 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:20.505069 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.502240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:36583 (local address 127.26.98.2:39307)
0619 02:49:20.502292 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.502293 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.502303 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:20.502336 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.502337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.502349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.502405 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.502562 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.503143 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504270 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.504681 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504754 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.504768 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.504926 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.504976 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:20.505115 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.502137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:39307 (local address 127.26.98.3:36583)
0619 02:49:20.502186 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.502199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.502257 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.502416 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.502417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.502419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.502501 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.502503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.503244 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.503246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.504169 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.504173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.504768 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.504861 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.504865 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.504914 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.504979 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":25}
I20260619 02:49:20.505395 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.502159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:35885 (local address 127.26.98.3:34429)
0619 02:49:20.502345 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.502355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.502426 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.502555 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.502556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.502557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.502637 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.502638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.503281 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.503282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.504171 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.504173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.504769 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.504769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.504860 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.504865 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.504922 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.504974 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":21,"thread_start_us":112,"threads_started":1}
I20260619 02:49:20.505606 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334a0e442db24f7a9bb9888487cb70e9" candidate_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bd29c7dfdc455caf31b9ce20f45127" is_pre_election: true
I20260619 02:49:20.505826 28428 raft_consensus.cc:2454] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2283255d7f4888ab3b538b76fd6fbb in term 0.
I20260619 02:49:20.505829 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334a0e442db24f7a9bb9888487cb70e9" candidate_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354d7b7156b74c7f93ad4a11938822b4" is_pre_election: true
I20260619 02:49:20.505968 28288 raft_consensus.cc:2454] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2283255d7f4888ab3b538b76fd6fbb in term 0.
I20260619 02:49:20.506165 28499 leader_election.cc:302] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354d7b7156b74c7f93ad4a11938822b4, cd2283255d7f4888ab3b538b76fd6fbb; no voters:
I20260619 02:49:20.506371 28645 raft_consensus.cc:2781] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:20.506417 28645 raft_consensus.cc:482] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:20.506433 28645 raft_consensus.cc:3037] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.507786 28645 raft_consensus.cc:504] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.507975 28645 leader_election.cc:288] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [CANDIDATE]: Term 1 election: Requested vote from peers 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885), 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307)
I20260619 02:49:20.508219 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334a0e442db24f7a9bb9888487cb70e9" candidate_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354d7b7156b74c7f93ad4a11938822b4"
I20260619 02:49:20.508286 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334a0e442db24f7a9bb9888487cb70e9" candidate_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bd29c7dfdc455caf31b9ce20f45127"
I20260619 02:49:20.508356 28288 raft_consensus.cc:3037] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.508369 28428 raft_consensus.cc:3037] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.509243 28428 raft_consensus.cc:2454] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2283255d7f4888ab3b538b76fd6fbb in term 1.
I20260619 02:49:20.509240 28288 raft_consensus.cc:2454] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2283255d7f4888ab3b538b76fd6fbb in term 1.
I20260619 02:49:20.509531 28499 leader_election.cc:302] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354d7b7156b74c7f93ad4a11938822b4, cd2283255d7f4888ab3b538b76fd6fbb; no voters:
I20260619 02:49:20.509675 28645 raft_consensus.cc:2781] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:20.509876 28645 raft_consensus.cc:686] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 1 LEADER]: Becoming Leader. State: Replica: cd2283255d7f4888ab3b538b76fd6fbb, State: Running, Role: LEADER
I20260619 02:49:20.510074 28645 consensus_queue.cc:237] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } }
I20260619 02:49:20.511382 28131 catalog_manager.cc:5495] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb reported cstate change: term changed from 0 to 1, leader changed from <none> to cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3). New cstate: current_term: 1 leader_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:20.547569 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.544739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.26.98.3:46175)
0619 02:49:20.544832 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.544835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.544843 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:20.544897 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.544899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.544926 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.544991 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:20.545165 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.545703 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.546585 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.546799 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.546861 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.547140 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.547278 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.547357 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.547403 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.547434 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:49:20.549929 28428 raft_consensus.cc:1261] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Refusing update from remote peer cd2283255d7f4888ab3b538b76fd6fbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.549965 28288 raft_consensus.cc:1261] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Refusing update from remote peer cd2283255d7f4888ab3b538b76fd6fbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.550414 28647 consensus_queue.cc:1035] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.550721 28645 consensus_queue.cc:1035] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.554049 28650 mvcc.cc:205] Tried to move back new op lower bound from 7298405828808994816 to 7298405828649955328. Current Snapshot: MvccSnapshot[applied={T|T < 7298405828808994816}]
I20260619 02:49:20.560245 28130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:20.560407 28130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:20.561510 28130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Sending DeleteTablet for 3 replicas of tablet 334a0e442db24f7a9bb9888487cb70e9
I20260619 02:49:20.562059 28268 tablet_service.cc:1514] Processing DeleteTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:34460
I20260619 02:49:20.562204 28548 tablet_service.cc:1514] Processing DeleteTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:56672
I20260619 02:49:20.562659 28660 tablet_replica.cc:330] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: stopping tablet replica
I20260619 02:49:20.562644 28408 tablet_service.cc:1514] Processing DeleteTablet for tablet 334a0e442db24f7a9bb9888487cb70e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:38564
I20260619 02:49:20.562662 28130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:49:20.562798 28660 raft_consensus.cc:2229] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 1 LEADER]: Raft consensus shutting down.
W20260619 02:49:20.562856 28130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:20.563170 28660 raft_consensus.cc:2258] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.563670 28659 tablet_replica.cc:330] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: stopping tablet replica
I20260619 02:49:20.563907 28660 ts_tablet_manager.cc:1905] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.564883 28268 tablet_service.cc:1467] Processing CreateTablet for tablet 39597c6d9a0b462f9e5e8c807331811f (DEFAULT_TABLE table=test-table [id=54b479959e3b4406b9938c93d36a66a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.565043 28268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39597c6d9a0b462f9e5e8c807331811f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.565043 28548 tablet_service.cc:1467] Processing CreateTablet for tablet 39597c6d9a0b462f9e5e8c807331811f (DEFAULT_TABLE table=test-table [id=54b479959e3b4406b9938c93d36a66a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.565153 28548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39597c6d9a0b462f9e5e8c807331811f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.565912 28660 ts_tablet_manager.cc:1918] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:20.565985 28660 log.cc:1196] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/334a0e442db24f7a9bb9888487cb70e9
I20260619 02:49:20.566227 28660 ts_tablet_manager.cc:1939] T 334a0e442db24f7a9bb9888487cb70e9 P cd2283255d7f4888ab3b538b76fd6fbb: Deleting consensus metadata
I20260619 02:49:20.566619 28637 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Bootstrap starting.
I20260619 02:49:20.566672 28638 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Bootstrap starting.
I20260619 02:49:20.567101 28637 tablet_bootstrap.cc:654] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.567111 28638 tablet_bootstrap.cc:654] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.567646 28118 catalog_manager.cc:4841] TS cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175): tablet 334a0e442db24f7a9bb9888487cb70e9 (table test-table [id=dcad63afca684eaea1fd9d8be53b744c]) successfully deleted
I20260619 02:49:20.567687 28408 tablet_service.cc:1467] Processing CreateTablet for tablet 39597c6d9a0b462f9e5e8c807331811f (DEFAULT_TABLE table=test-table [id=54b479959e3b4406b9938c93d36a66a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:20.567837 28408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39597c6d9a0b462f9e5e8c807331811f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.568015 28638 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: No bootstrap required, opened a new log
I20260619 02:49:20.568063 28638 ts_tablet_manager.cc:1397] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.568239 28638 raft_consensus.cc:348] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.568310 28638 raft_consensus.cc:374] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.568322 28638 raft_consensus.cc:729] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354d7b7156b74c7f93ad4a11938822b4, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.568372 28638 consensus_queue.cc:260] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.568490 28638 ts_tablet_manager.cc:1428] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.569324 28639 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Bootstrap starting.
I20260619 02:49:20.569810 28639 tablet_bootstrap.cc:654] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.569841 28637 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: No bootstrap required, opened a new log
I20260619 02:49:20.569890 28637 ts_tablet_manager.cc:1397] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:20.570102 28637 raft_consensus.cc:348] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.570233 28637 raft_consensus.cc:374] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.570258 28637 raft_consensus.cc:729] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2283255d7f4888ab3b538b76fd6fbb, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.570308 28637 consensus_queue.cc:260] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.570418 28637 ts_tablet_manager.cc:1428] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:20.563898 28659 raft_consensus.cc:2229] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:20.570760 28639 tablet_bootstrap.cc:492] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: No bootstrap required, opened a new log
I20260619 02:49:20.570822 28639 ts_tablet_manager.cc:1397] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:20.571051 28659 raft_consensus.cc:2258] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.571081 28639 raft_consensus.cc:348] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.571182 28639 raft_consensus.cc:374] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.571210 28639 raft_consensus.cc:729] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bd29c7dfdc455caf31b9ce20f45127, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.571255 28639 consensus_queue.cc:260] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.571395 28639 ts_tablet_manager.cc:1428] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.572116 28659 ts_tablet_manager.cc:1905] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.573861 28661 tablet_replica.cc:330] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: stopping tablet replica
I20260619 02:49:20.573918 28659 ts_tablet_manager.cc:1918] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:20.573999 28659 log.cc:1196] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/334a0e442db24f7a9bb9888487cb70e9
I20260619 02:49:20.574062 28661 raft_consensus.cc:2229] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:20.574232 28661 raft_consensus.cc:2258] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.574661 28659 ts_tablet_manager.cc:1939] T 334a0e442db24f7a9bb9888487cb70e9 P 354d7b7156b74c7f93ad4a11938822b4: Deleting consensus metadata
I20260619 02:49:20.574951 28661 ts_tablet_manager.cc:1905] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.575244 28115 catalog_manager.cc:4841] TS 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885): tablet 334a0e442db24f7a9bb9888487cb70e9 (table test-table [id=dcad63afca684eaea1fd9d8be53b744c]) successfully deleted
I20260619 02:49:20.576467 28661 ts_tablet_manager.cc:1918] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:20.576550 28661 log.cc:1196] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/334a0e442db24f7a9bb9888487cb70e9
I20260619 02:49:20.577081 28661 ts_tablet_manager.cc:1939] T 334a0e442db24f7a9bb9888487cb70e9 P 83bd29c7dfdc455caf31b9ce20f45127: Deleting consensus metadata
I20260619 02:49:20.577764 28118 catalog_manager.cc:4841] TS 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307): tablet 334a0e442db24f7a9bb9888487cb70e9 (table test-table [id=dcad63afca684eaea1fd9d8be53b744c]) successfully deleted
I20260619 02:49:20.603806 28643 raft_consensus.cc:482] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:20.603948 28643 raft_consensus.cc:504] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.604377 28643 leader_election.cc:288] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885), cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175)
I20260619 02:49:20.608559 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.604579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:46175 (local address 127.26.98.2:36453)
0619 02:49:20.604798 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.604815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.604904 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.605071 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.605072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.605074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.605174 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.605177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.606131 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.606132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.607484 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.607490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.608216 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.608217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.608307 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.608312 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.608365 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.608426 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":29,"thread_start_us":78,"threads_started":1}
I20260619 02:49:20.608528 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.604776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:36453 (local address 127.26.98.3:46175)
0619 02:49:20.604825 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.604828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.604886 (+ 58us) server_negotiation.cc:407] Connection header received
0619 02:49:20.604982 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.604984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.604994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.605046 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.605282 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.606036 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.607653 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.608117 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.608199 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.608226 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.608380 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.608428 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:20.608984 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.604795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:52569 (local address 127.26.98.1:35885)
0619 02:49:20.604840 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.604842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.604959 (+ 117us) server_negotiation.cc:407] Connection header received
0619 02:49:20.605079 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.605081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.605092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.605149 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.605344 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.605875 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.607655 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.608329 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.608409 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.608422 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.608840 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.608898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:20.608933 28568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39597c6d9a0b462f9e5e8c807331811f" candidate_uuid: "83bd29c7dfdc455caf31b9ce20f45127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" is_pre_election: true
I20260619 02:49:20.609092 28568 raft_consensus.cc:2454] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bd29c7dfdc455caf31b9ce20f45127 in term 0.
I20260619 02:49:20.609248 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.604579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:35885 (local address 127.26.98.2:52569)
0619 02:49:20.604886 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.604896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.604984 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.605161 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.605162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.605163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.605219 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.605221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.605991 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.605994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.607484 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.607490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.608486 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.608488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.608578 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.608585 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.609092 (+ 507us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.609156 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":34,"thread_start_us":76,"threads_started":1}
I20260619 02:49:20.609329 28362 leader_election.cc:302] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83bd29c7dfdc455caf31b9ce20f45127, cd2283255d7f4888ab3b538b76fd6fbb; no voters:
I20260619 02:49:20.609495 28643 raft_consensus.cc:2781] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:20.609541 28643 raft_consensus.cc:482] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:20.609562 28643 raft_consensus.cc:3037] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.609638 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39597c6d9a0b462f9e5e8c807331811f" candidate_uuid: "83bd29c7dfdc455caf31b9ce20f45127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354d7b7156b74c7f93ad4a11938822b4" is_pre_election: true
I20260619 02:49:20.609774 28288 raft_consensus.cc:2454] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bd29c7dfdc455caf31b9ce20f45127 in term 0.
I20260619 02:49:20.610538 28643 raft_consensus.cc:504] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.610703 28643 leader_election.cc:288] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [CANDIDATE]: Term 1 election: Requested vote from peers 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885), cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175)
I20260619 02:49:20.610911 28288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39597c6d9a0b462f9e5e8c807331811f" candidate_uuid: "83bd29c7dfdc455caf31b9ce20f45127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354d7b7156b74c7f93ad4a11938822b4"
I20260619 02:49:20.611006 28288 raft_consensus.cc:3037] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.611011 28568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39597c6d9a0b462f9e5e8c807331811f" candidate_uuid: "83bd29c7dfdc455caf31b9ce20f45127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2283255d7f4888ab3b538b76fd6fbb"
I20260619 02:49:20.611096 28568 raft_consensus.cc:3037] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.611994 28568 raft_consensus.cc:2454] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bd29c7dfdc455caf31b9ce20f45127 in term 1.
I20260619 02:49:20.611992 28288 raft_consensus.cc:2454] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bd29c7dfdc455caf31b9ce20f45127 in term 1.
I20260619 02:49:20.612264 28359 leader_election.cc:302] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354d7b7156b74c7f93ad4a11938822b4, 83bd29c7dfdc455caf31b9ce20f45127; no voters:
I20260619 02:49:20.612381 28643 raft_consensus.cc:2781] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:20.612588 28643 raft_consensus.cc:686] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 LEADER]: Becoming Leader. State: Replica: 83bd29c7dfdc455caf31b9ce20f45127, State: Running, Role: LEADER
I20260619 02:49:20.612686 28643 consensus_queue.cc:237] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.613742 28130 catalog_manager.cc:5495] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "83bd29c7dfdc455caf31b9ce20f45127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837359, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:20.672621 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.663950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.26.98.62:35193)
0619 02:49:20.664024 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.664026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.664105 (+ 79us) server_negotiation.cc:407] Connection header received
0619 02:49:20.664196 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.664197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.664210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.664293 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.664905 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.665672 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.666475 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.666767 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.666848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.669594 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.669596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.669596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.669604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.671039 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.671575 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.671577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.671578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.671648 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.671896 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.671897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.671902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.672048 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.672114 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.672337 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.672502 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":40}
I20260619 02:49:20.672638 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.663866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35193 (local address 127.0.0.1:37596)
0619 02:49:20.664035 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.664049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.664135 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.664357 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.664358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.664757 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:20.664819 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.664822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.665789 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.665790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.666390 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.666392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.666871 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.666872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.666988 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.667716 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:20.667720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:20.669436 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:20.671121 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.671135 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.671139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.671467 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.671724 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:20.671725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:20.671725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:20.671798 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:20.672114 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:20.672123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:20.672211 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.672286 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.672450 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260619 02:49:20.675627 28130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:37596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:49:20.675907 28130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:20.677621 28268 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf06cb095b24b259760aa74f2aefe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93dbd1de332c4335b3fffcdf3700c683]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:20.677812 28268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf06cb095b24b259760aa74f2aefe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.679126 28638 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: Bootstrap starting.
I20260619 02:49:20.679093 28548 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf06cb095b24b259760aa74f2aefe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93dbd1de332c4335b3fffcdf3700c683]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:20.679272 28548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf06cb095b24b259760aa74f2aefe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.679564 28638 tablet_bootstrap.cc:654] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.680171 28638 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: No bootstrap required, opened a new log
I20260619 02:49:20.680209 28638 ts_tablet_manager.cc:1397] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.680347 28638 raft_consensus.cc:348] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.680450 28638 raft_consensus.cc:374] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.680470 28638 raft_consensus.cc:729] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354d7b7156b74c7f93ad4a11938822b4, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.680519 28638 consensus_queue.cc:260] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.680622 28638 ts_tablet_manager.cc:1428] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.680685 28644 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.681103 28637 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: Bootstrap starting.
I20260619 02:49:20.681102 28644 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.681242 28408 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf06cb095b24b259760aa74f2aefe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93dbd1de332c4335b3fffcdf3700c683]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:20.681403 28408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf06cb095b24b259760aa74f2aefe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.681628 28637 tablet_bootstrap.cc:654] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.682482 28637 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: No bootstrap required, opened a new log
I20260619 02:49:20.682521 28637 ts_tablet_manager.cc:1397] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:20.682689 28637 raft_consensus.cc:348] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.682790 28637 raft_consensus.cc:374] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.682806 28637 raft_consensus.cc:729] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2283255d7f4888ab3b538b76fd6fbb, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.682847 28637 consensus_queue.cc:260] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.683097 28637 ts_tablet_manager.cc:1428] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:20.683198 28647 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.683283 28647 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.683388 28639 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: Bootstrap starting.
I20260619 02:49:20.683867 28639 tablet_bootstrap.cc:654] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.684621 28639 tablet_bootstrap.cc:492] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: No bootstrap required, opened a new log
I20260619 02:49:20.684681 28639 ts_tablet_manager.cc:1397] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.684847 28639 raft_consensus.cc:348] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.684942 28639 raft_consensus.cc:374] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.684962 28639 raft_consensus.cc:729] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bd29c7dfdc455caf31b9ce20f45127, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.685132 28639 consensus_queue.cc:260] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.685592 28639 ts_tablet_manager.cc:1428] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:20.686378 28643 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.686481 28643 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.770738 28644 raft_consensus.cc:482] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:20.770902 28644 raft_consensus.cc:504] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.771313 28644 leader_election.cc:288] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307), cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175)
I20260619 02:49:20.775492 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.771685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:56613 (local address 127.26.98.2:39307)
0619 02:49:20.771774 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.771776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.771809 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:20.771904 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.771905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.771916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.771981 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.772201 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.772809 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.774341 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.774833 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.774950 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.774967 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.775319 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.775403 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:49:20.775584 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.771563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:39307 (local address 127.26.98.1:56613)
0619 02:49:20.771708 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.771722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.771802 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.771996 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.771997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.772001 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.772099 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.772102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.772935 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.772936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.774134 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.774141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.775018 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.775020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.775113 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.775122 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.775222 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.775305 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":41,"thread_start_us":84,"threads_started":1}
I20260619 02:49:20.775695 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.771859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:41491 (local address 127.26.98.3:46175)
0619 02:49:20.771916 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.771918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.772000 (+ 82us) server_negotiation.cc:407] Connection header received
0619 02:49:20.772062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.772063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.772076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.772152 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:20.772376 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.772923 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.774552 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.775129 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.775263 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.775278 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.775560 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.775624 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:20.775735 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.771786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:46175 (local address 127.26.98.1:41491)
0619 02:49:20.771935 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.771943 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.771997 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.772197 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.772198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.772202 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:20.772287 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.772289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.773055 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.773056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.774363 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.774373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.775323 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.775325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.775415 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.775420 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.775495 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.775548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260619 02:49:20.776007 28568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf06cb095b24b259760aa74f2aefe4c" candidate_uuid: "354d7b7156b74c7f93ad4a11938822b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" is_pre_election: true
I20260619 02:49:20.776010 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf06cb095b24b259760aa74f2aefe4c" candidate_uuid: "354d7b7156b74c7f93ad4a11938822b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bd29c7dfdc455caf31b9ce20f45127" is_pre_election: true
I20260619 02:49:20.776118 28428 raft_consensus.cc:2454] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354d7b7156b74c7f93ad4a11938822b4 in term 0.
I20260619 02:49:20.776119 28568 raft_consensus.cc:2454] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354d7b7156b74c7f93ad4a11938822b4 in term 0.
I20260619 02:49:20.776326 28222 leader_election.cc:302] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354d7b7156b74c7f93ad4a11938822b4, cd2283255d7f4888ab3b538b76fd6fbb; no voters:
I20260619 02:49:20.776507 28644 raft_consensus.cc:2781] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:20.776557 28644 raft_consensus.cc:482] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:20.776571 28644 raft_consensus.cc:3037] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.777482 28644 raft_consensus.cc:504] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.777791 28644 leader_election.cc:288] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [CANDIDATE]: Term 1 election: Requested vote from peers 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307), cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175)
I20260619 02:49:20.778219 28428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf06cb095b24b259760aa74f2aefe4c" candidate_uuid: "354d7b7156b74c7f93ad4a11938822b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bd29c7dfdc455caf31b9ce20f45127"
I20260619 02:49:20.778208 28568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf06cb095b24b259760aa74f2aefe4c" candidate_uuid: "354d7b7156b74c7f93ad4a11938822b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2283255d7f4888ab3b538b76fd6fbb"
I20260619 02:49:20.778330 28428 raft_consensus.cc:3037] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.778330 28568 raft_consensus.cc:3037] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.779350 28428 raft_consensus.cc:2454] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354d7b7156b74c7f93ad4a11938822b4 in term 1.
I20260619 02:49:20.779345 28568 raft_consensus.cc:2454] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354d7b7156b74c7f93ad4a11938822b4 in term 1.
I20260619 02:49:20.779596 28222 leader_election.cc:302] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354d7b7156b74c7f93ad4a11938822b4, cd2283255d7f4888ab3b538b76fd6fbb; no voters:
I20260619 02:49:20.779734 28644 raft_consensus.cc:2781] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:20.779958 28644 raft_consensus.cc:686] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [term 1 LEADER]: Becoming Leader. State: Replica: 354d7b7156b74c7f93ad4a11938822b4, State: Running, Role: LEADER
I20260619 02:49:20.780056 28644 consensus_queue.cc:237] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } }
I20260619 02:49:20.780596 28680 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: TxnStatusTablet state changed. Reason: New leader 354d7b7156b74c7f93ad4a11938822b4. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.780720 28680 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:20.780953 28682 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:49:20.781070 28682 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:49:20.781112 28130 catalog_manager.cc:5495] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1). New cstate: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:20.792786 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.789815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:35885 (local address 127.0.0.1:34462)
0619 02:49:20.789905 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.789918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.789994 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.790192 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.790193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.790194 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:20.790297 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.790301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.791070 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.791071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.791728 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.791731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.792125 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.792126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.792186 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.792221 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.792551 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.792563 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.792618 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.792702 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 02:49:20.792786 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.789784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.26.98.1:35885)
0619 02:49:20.789846 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.789849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.789973 (+ 124us) server_negotiation.cc:407] Connection header received
0619 02:49:20.790087 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.790088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.790112 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.790172 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:20.790364 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.790984 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.791831 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.792039 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.792112 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.792334 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.792489 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.792564 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.792625 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.792652 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:20.794942 28568 raft_consensus.cc:1261] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Refusing update from remote peer 354d7b7156b74c7f93ad4a11938822b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.794950 28428 raft_consensus.cc:1261] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Refusing update from remote peer 354d7b7156b74c7f93ad4a11938822b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.795353 28680 consensus_queue.cc:1035] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.795562 28644 consensus_queue.cc:1035] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.796329 28647 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: TxnStatusTablet state changed. Reason: New leader 354d7b7156b74c7f93ad4a11938822b4. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.796344 28643 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: TxnStatusTablet state changed. Reason: New leader 354d7b7156b74c7f93ad4a11938822b4. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.796454 28647 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.796455 28643 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.797443 28644 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.797560 28644 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:20.797923 28652 mvcc.cc:205] Tried to move back new op lower bound from 7298405829812674560 to 7298405829756252160. Current Snapshot: MvccSnapshot[applied={T|T < 7298405829812674560 or (T in {7298405829812674560})}]
I20260619 02:49:20.798641 28647 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.798753 28647 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P cd2283255d7f4888ab3b538b76fd6fbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.802970 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.799872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.26.98.2:39307)
0619 02:49:20.799918 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.799920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.799930 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:20.800026 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.800027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.800048 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.800116 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:20.800259 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.800785 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.801974 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.802191 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.802266 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.802443 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.802586 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.802648 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.802704 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.802727 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:20.803812 28680 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.803954 28680 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 354d7b7156b74c7f93ad4a11938822b4: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:20.804769 28643 tablet_replica.cc:439] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "354d7b7156b74c7f93ad4a11938822b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 } } peers { permanent_uuid: "83bd29c7dfdc455caf31b9ce20f45127" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 39307 } } peers { permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 } } }
I20260619 02:49:20.804908 28643 tablet_replica.cc:442] T 4cf06cb095b24b259760aa74f2aefe4c P 83bd29c7dfdc455caf31b9ce20f45127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:20.809556 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.806504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:45511 (local address 127.26.98.1:35885)
0619 02:49:20.806559 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.806561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.806724 (+ 163us) server_negotiation.cc:407] Connection header received
0619 02:49:20.806796 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.806797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.806810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.806864 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:20.807067 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.807689 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.808781 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.808987 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.809045 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.809288 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.809328 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.809372 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.809426 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.809453 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:20.809679 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.806420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:35885 (local address 127.26.98.2:45511)
0619 02:49:20.806646 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.806657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.806725 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.806868 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.806870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.806871 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:20.806998 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.807000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.807783 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.807785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.808675 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.808680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.809055 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.809056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.809137 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.809164 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.809378 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.809380 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.809423 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.809486 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260619 02:49:20.812711 28288 raft_consensus.cc:1261] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Refusing update from remote peer 83bd29c7dfdc455caf31b9ce20f45127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.812741 28568 raft_consensus.cc:1261] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Refusing update from remote peer 83bd29c7dfdc455caf31b9ce20f45127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:20.813114 28667 consensus_queue.cc:1035] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "354d7b7156b74c7f93ad4a11938822b4" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.813918 28667 consensus_queue.cc:1035] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2283255d7f4888ab3b538b76fd6fbb" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:20.815368 28651 mvcc.cc:205] Tried to move back new op lower bound from 7298405829886291968 to 7298405829070123008. Current Snapshot: MvccSnapshot[applied={T|T < 7298405829886291968}]
I20260619 02:49:20.827503 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.823771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:54207 (local address 127.26.98.2:39307)
0619 02:49:20.823849 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.823851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.824023 (+ 172us) server_negotiation.cc:407] Connection header received
0619 02:49:20.824097 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.824098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.824110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.824162 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:20.824413 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.825163 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.826593 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.826854 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.826981 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.827200 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.827237 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.827277 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.827359 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.827389 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:49:20.827551 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.823702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:39307 (local address 127.26.98.1:54207)
0619 02:49:20.823930 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.823945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.824032 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.824168 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.824169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.824171 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:20.824263 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.824265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.825310 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.825311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.826458 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.826465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.826980 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.826981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.827064 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.827104 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:20.827305 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:20.827308 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.827358 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.827421 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":134,"threads_started":1}
I20260619 02:49:20.842629 28130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:20.842787 28130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:20.843622 28130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eaf68fbce214bfebcf6aaa731cf81b5: Sending DeleteTablet for 3 replicas of tablet 39597c6d9a0b462f9e5e8c807331811f
I20260619 02:49:20.844031 28548 tablet_service.cc:1514] Processing DeleteTablet for tablet 39597c6d9a0b462f9e5e8c807331811f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:56672
I20260619 02:49:20.844084 28408 tablet_service.cc:1514] Processing DeleteTablet for tablet 39597c6d9a0b462f9e5e8c807331811f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:38564
I20260619 02:49:20.844187 28661 tablet_replica.cc:330] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: stopping tablet replica
I20260619 02:49:20.844189 28268 tablet_service.cc:1514] Processing DeleteTablet for tablet 39597c6d9a0b462f9e5e8c807331811f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:34460
I20260619 02:49:20.844195 28660 tablet_replica.cc:330] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: stopping tablet replica
I20260619 02:49:20.844274 28661 raft_consensus.cc:2229] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:20.844310 28660 raft_consensus.cc:2229] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:20.844381 28660 raft_consensus.cc:2258] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.844409 28661 raft_consensus.cc:2258] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.844494 28659 tablet_replica.cc:330] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: stopping tablet replica
I20260619 02:49:20.844574 28659 raft_consensus.cc:2229] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:20.844635 28659 raft_consensus.cc:2258] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:20.845402 28659 ts_tablet_manager.cc:1905] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.845877 28661 ts_tablet_manager.cc:1905] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.847752 28661 ts_tablet_manager.cc:1918] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:20.847752 28659 ts_tablet_manager.cc:1918] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:20.847846 28661 log.cc:1196] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/39597c6d9a0b462f9e5e8c807331811f
I20260619 02:49:20.847855 28659 log.cc:1196] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/39597c6d9a0b462f9e5e8c807331811f
I20260619 02:49:20.848068 28659 ts_tablet_manager.cc:1939] T 39597c6d9a0b462f9e5e8c807331811f P 354d7b7156b74c7f93ad4a11938822b4: Deleting consensus metadata
I20260619 02:49:20.848075 28661 ts_tablet_manager.cc:1939] T 39597c6d9a0b462f9e5e8c807331811f P 83bd29c7dfdc455caf31b9ce20f45127: Deleting consensus metadata
I20260619 02:49:20.848470 28115 catalog_manager.cc:4841] TS 354d7b7156b74c7f93ad4a11938822b4 (127.26.98.1:35885): tablet 39597c6d9a0b462f9e5e8c807331811f (table test-table [id=54b479959e3b4406b9938c93d36a66a4]) successfully deleted
I20260619 02:49:20.848470 28118 catalog_manager.cc:4841] TS 83bd29c7dfdc455caf31b9ce20f45127 (127.26.98.2:39307): tablet 39597c6d9a0b462f9e5e8c807331811f (table test-table [id=54b479959e3b4406b9938c93d36a66a4]) successfully deleted
I20260619 02:49:20.848824 28660 ts_tablet_manager.cc:1905] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:20.851183 28660 ts_tablet_manager.cc:1918] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:20.851293 28660 log.cc:1196] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/39597c6d9a0b462f9e5e8c807331811f
I20260619 02:49:20.851588 28660 ts_tablet_manager.cc:1939] T 39597c6d9a0b462f9e5e8c807331811f P cd2283255d7f4888ab3b538b76fd6fbb: Deleting consensus metadata
I20260619 02:49:20.852169 28118 catalog_manager.cc:4841] TS cd2283255d7f4888ab3b538b76fd6fbb (127.26.98.3:46175): tablet 39597c6d9a0b462f9e5e8c807331811f (table test-table [id=54b479959e3b4406b9938c93d36a66a4]) successfully deleted
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:20.859140 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.847552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.26.98.1:35885)
0619 02:49:20.847640 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.847643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.847922 (+ 279us) server_negotiation.cc:407] Connection header received
0619 02:49:20.848031 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.848032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.848044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.848075 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.848891 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.849453 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.850292 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.850509 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.850580 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.854684 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.854700 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.854701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.854711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.857584 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.858121 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.858127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.858127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.858210 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.858416 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.858417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.858418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.858608 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.858802 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.858844 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.859004 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:49:20.859653 28328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34474
I20260619 02:49:20.869009 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.861666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.26.98.62:35193)
0619 02:49:20.861739 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.861742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.862061 (+ 319us) server_negotiation.cc:407] Connection header received
0619 02:49:20.862130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.862132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.862144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.862198 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.862845 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.863452 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.864433 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.864647 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.864721 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.866258 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.866260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.866260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:20.866271 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.867718 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.868086 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.868087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.868087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.868148 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:20.868321 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:20.868322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:20.868322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:20.868458 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.868557 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.868777 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.868896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 02:49:20.870008 28130 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37600
I20260619 02:49:20.871335 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28203
I20260619 02:49:20.880259 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28343
I20260619 02:49:20.885669 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28483
I20260619 02:49:20.891978 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28099
2026-06-19T02:49:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3785 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28704](info): set up 2 sockets
Jun 19 02:49:20 dist-test-slave-0f9d krb5kdc[28704](info): commencing operation
krb5kdc: starting...
W20260619 02:49:21.887315 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:21 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:22.027329 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:42593
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:22.129114 28717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:22.129498 28717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:22.129546 28717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:22.131891 28717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:22.132012 28717 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:49:22.132036 28717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:22.132055 28717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:22.132074 28717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:22.132090 28717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:22.135159 28717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:42593
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:22.135587 28717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:22.135906 28717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:22.143322 28723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.143321 28725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:22.143374 28717 server_base.cc:1034] running on GCE node
W20260619 02:49:22.143349 28722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:22.143951 28717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:22.144397 28717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:22.145632 28717 hybrid_clock.cc:648] HybridClock initialized: now 1781837362145582 us; error 78 us; skew 500 ppm
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:22.149888 28717 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:22.150633 28717 webserver.cc:460] Webserver started at http://127.26.98.62:35823/ using document root <none> and password file <none>
I20260619 02:49:22.150949 28717 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:22.151017 28717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:22.151158 28717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:22.152926 28717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "a61b9af13c224a4a9253908418bbf0a4"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "2201862b5486621a717e260265b8d6a4"
server_key_iv: "39febd639a69125dcb9c1aef7cd82192"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.153450 28717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "a61b9af13c224a4a9253908418bbf0a4"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "2201862b5486621a717e260265b8d6a4"
server_key_iv: "39febd639a69125dcb9c1aef7cd82192"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.155275 28717 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:49:22.156270 28732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.156476 28717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.156579 28717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "a61b9af13c224a4a9253908418bbf0a4"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "2201862b5486621a717e260265b8d6a4"
server_key_iv: "39febd639a69125dcb9c1aef7cd82192"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.156661 28717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:22.186334 28717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:22.189142 28717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:22.189412 28717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:22.195290 28717 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:42593
I20260619 02:49:22.195384 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:42593 every 8 connection(s)
I20260619 02:49:22.195791 28717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:22.196872 28795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.200574 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Bootstrap starting.
I20260619 02:49:22.201375 28795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.201705 28795 log.cc:824] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:22.202701 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: No bootstrap required, opened a new log
I20260619 02:49:22.205230 28795 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } }
I20260619 02:49:22.205471 28795 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.205518 28795 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61b9af13c224a4a9253908418bbf0a4, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.205528 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28717
I20260619 02:49:22.205654 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:22.205725 28795 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } }
I20260619 02:49:22.205847 27016 external_mini_cluster.cc:1428] Setting key 082bac017eac48305b540c284f92fc8e
I20260619 02:49:22.205840 28795 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:22.205892 28795 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:22.205955 28795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:22.206976 28795 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } }
I20260619 02:49:22.207139 28795 leader_election.cc:302] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61b9af13c224a4a9253908418bbf0a4; no voters:
I20260619 02:49:22.207361 28795 leader_election.cc:288] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:22.207441 28800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:22.207695 28800 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [term 1 LEADER]: Becoming Leader. State: Replica: a61b9af13c224a4a9253908418bbf0a4, State: Running, Role: LEADER
I20260619 02:49:22.207743 28795 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:22.207831 28800 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } }
I20260619 02:49:22.208276 28800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61b9af13c224a4a9253908418bbf0a4. Latest consensus state: current_term: 1 leader_uuid: "a61b9af13c224a4a9253908418bbf0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } } }
I20260619 02:49:22.208338 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61b9af13c224a4a9253908418bbf0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61b9af13c224a4a9253908418bbf0a4" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42593 } } }
I20260619 02:49:22.208395 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:22.208364 28800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:22.209355 28808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:22.210091 28808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:22.212670 28808 catalog_manager.cc:1347] Generated new cluster ID: c73ff92ec36045c8b7037b39d944fab5
I20260619 02:49:22.212759 28808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.223055 28808 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:22.223093 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.206655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.26.98.62:42593)
0619 02:49:22.206966 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.206982 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.207120 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:49:22.207312 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.207322 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.207399 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.208490 (+ 1091us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.208970 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.210695 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.211944 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.212662 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.212819 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.217266 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.217280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.217281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.217310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.220059 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.220919 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.220927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.220927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.221151 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.221399 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.221400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.221401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.221843 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.222070 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.222237 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.222701 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":113,"threads_started":1}
I20260619 02:49:22.224174 28808 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:22.232576 28808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Generated new TSK 0
I20260619 02:49:22.232944 28808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:22.268441 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:42593
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:22.369855 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:22.370165 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:22.370211 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:22.371770 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:22.371846 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:22.371881 28821 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:22.371896 28821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:22.371922 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:22.373867 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:22.374209 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:22.374483 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:22.380409 28826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.380415 28829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.380415 28827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:22.380662 28821 server_base.cc:1034] running on GCE node
I20260619 02:49:22.380971 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:22.381299 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:22.382634 28821 hybrid_clock.cc:648] HybridClock initialized: now 1781837362382576 us; error 71 us; skew 500 ppm
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:22.386313 28821 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:22.387015 28821 webserver.cc:460] Webserver started at http://127.26.98.1:42213/ using document root <none> and password file <none>
I20260619 02:49:22.387276 28821 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:22.387326 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:22.387471 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:22.388749 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "1e9e12f461c745428b9db89adbfb0a46"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "bea123ee7fab30ba165e8ba5829a294b"
server_key_iv: "3675be516a27dfa644abe7ae8f5c74c4"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.389132 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "1e9e12f461c745428b9db89adbfb0a46"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "bea123ee7fab30ba165e8ba5829a294b"
server_key_iv: "3675be516a27dfa644abe7ae8f5c74c4"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.390689 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:22.391628 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.391883 28821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.391970 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "1e9e12f461c745428b9db89adbfb0a46"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "bea123ee7fab30ba165e8ba5829a294b"
server_key_iv: "3675be516a27dfa644abe7ae8f5c74c4"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.392057 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:22.411080 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:22.414100 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:22.414337 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:22.415064 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:22.415112 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.415146 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:22.415163 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.424226 28821 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:41301
I20260619 02:49:22.424394 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:41301 every 8 connection(s)
I20260619 02:49:22.424855 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:22.425935 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28821
I20260619 02:49:22.426029 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:22.426231 27016 external_mini_cluster.cc:1428] Setting key 948b09c455811a903c74a18fa8b00361
I20260619 02:49:22.428401 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.415674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:50691 (local address 127.26.98.62:42593)
0619 02:49:22.415737 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.415740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.416010 (+ 270us) server_negotiation.cc:407] Connection header received
0619 02:49:22.416682 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.416683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.416715 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.416794 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.418081 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.418988 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.420782 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.421130 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.421220 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.424618 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.424622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.424622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.424642 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.426477 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.427128 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.427129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.427130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.427216 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.427550 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.427551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.427551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.427729 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.427810 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.428176 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.428294 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:49:22.428567 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.415673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.1:50691)
0619 02:49:22.415869 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.415914 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.416546 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.416905 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.416911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.417363 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.417913 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.417923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.419106 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.419107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.420646 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.420650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.421231 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.421232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.421424 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.422087 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.422098 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.424423 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.426598 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.426607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.426611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.427036 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.427319 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.427320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.427321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.427472 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.427838 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.427845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.428030 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.428145 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.428324 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260619 02:49:22.429267 28952 heartbeater.cc:347] Connected to a master server at 127.26.98.62:42593
I20260619 02:49:22.429370 28952 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:22.429591 28952 heartbeater.cc:511] Master 127.26.98.62:42593 requested a full tablet report, sending...
I20260619 02:49:22.430198 28749 ts_manager.cc:194] Registered new tserver with Master: 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301)
I20260619 02:49:22.431042 28749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:50691
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:49:22.439507 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.430607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:37027 (local address 127.26.98.62:42593)
0619 02:49:22.430656 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.430658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.430804 (+ 146us) server_negotiation.cc:407] Connection header received
0619 02:49:22.430860 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.430861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.430871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.430952 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.431369 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.432303 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.433570 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.433794 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.433869 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.435489 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.435491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.435492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.435503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.437979 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.438466 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.438468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.438468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.438541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.438818 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.438819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.438819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.439028 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.439105 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.439296 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.439401 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:22.439509 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.430528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.1:37027)
0619 02:49:22.430725 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.430735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.430799 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.430960 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.430961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.431185 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.431268 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.431270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.432614 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.432615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.433428 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.433434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.433875 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.433876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.433972 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.434558 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.434564 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.435217 (+ 653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.438132 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.438134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.438134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.438367 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.438642 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.438643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.438643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.438746 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.439120 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.439121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.439181 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.439260 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.439386 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:22.460131 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:42593
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:22.560860 28961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:22.561208 28961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:22.561259 28961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:22.563024 28961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:22.563102 28961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:22.563130 28961 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:22.563141 28961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:22.563161 28961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:22.565598 28961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:22.565984 28961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:22.566252 28961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:22.572878 28967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.572885 28969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.572901 28966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:22.573323 28961 server_base.cc:1034] running on GCE node
I20260619 02:49:22.573602 28961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:22.573906 28961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:22.575071 28961 hybrid_clock.cc:648] HybridClock initialized: now 1781837362575046 us; error 45 us; skew 500 ppm
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:22.578312 28961 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:22.579000 28961 webserver.cc:460] Webserver started at http://127.26.98.2:40585/ using document root <none> and password file <none>
I20260619 02:49:22.579291 28961 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:22.579382 28961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:22.579545 28961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:22.580868 28961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "be2aaa42aff34448b4ee909dd46b2a54"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "7aeaedf27f501c6568b157f408319eb1"
server_key_iv: "7ce025bfcc5e7bd3bd07fb5ab9c1a830"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.581296 28961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "be2aaa42aff34448b4ee909dd46b2a54"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "7aeaedf27f501c6568b157f408319eb1"
server_key_iv: "7ce025bfcc5e7bd3bd07fb5ab9c1a830"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.582998 28961 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:49:22.584013 28976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.584280 28961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.584398 28961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "be2aaa42aff34448b4ee909dd46b2a54"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "7aeaedf27f501c6568b157f408319eb1"
server_key_iv: "7ce025bfcc5e7bd3bd07fb5ab9c1a830"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.584506 28961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:22.599107 28961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:22.601621 28961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:22.601795 28961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:22.602455 28961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:22.602617 28961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.602689 28961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:22.602716 28961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.611743 28961 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:44737
I20260619 02:49:22.611864 29091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:44737 every 8 connection(s)
I20260619 02:49:22.612195 28961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:22.614861 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.603016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:51623 (local address 127.26.98.62:42593)
0619 02:49:22.603101 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.603103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.603404 (+ 301us) server_negotiation.cc:407] Connection header received
0619 02:49:22.604352 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.604355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.604369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.604446 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.605677 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.606249 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.607783 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.608012 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.608089 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.611708 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.611711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.611712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.611722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.613211 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.613656 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.613665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.613666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.613726 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.613966 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.613967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.613968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.614102 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.614162 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.614482 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.614581 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:49:22.615160 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.603038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.2:51623)
0619 02:49:22.603225 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.603281 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.604165 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.604579 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.604585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.605076 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.605528 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.605538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.606391 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.606392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.607656 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.607662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.608112 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.608113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.608295 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.609169 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.609181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.611564 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.613310 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.613318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.613321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.613571 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.613783 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.613784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.613784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.613902 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.614168 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.614175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.614339 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.614437 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.614646 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260619 02:49:22.615824 29092 heartbeater.cc:347] Connected to a master server at 127.26.98.62:42593
I20260619 02:49:22.615944 29092 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:22.616196 29092 heartbeater.cc:511] Master 127.26.98.62:42593 requested a full tablet report, sending...
I20260619 02:49:22.616662 28749 ts_manager.cc:194] Registered new tserver with Master: be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:22.617123 28749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:51623
I20260619 02:49:22.617537 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 28961
I20260619 02:49:22.617756 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:22.618235 27016 external_mini_cluster.cc:1428] Setting key 50c0c7d8557a364f429b7dde221bb49b
I20260619 02:49:22.624532 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.617298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.2:37451)
0619 02:49:22.617684 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.617701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.617802 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.617929 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.617930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.618158 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.618256 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.618258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.619178 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.619179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.619744 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.619748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.620142 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.620142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.620235 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.620808 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.620813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.621197 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.622777 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.622778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.622779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.623032 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.623390 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.623391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.623391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.623470 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.624079 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.624080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.624165 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.624240 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.624392 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260619 02:49:22.624521 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.617414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:37451 (local address 127.26.98.62:42593)
0619 02:49:22.617470 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.617472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.617822 (+ 350us) server_negotiation.cc:407] Connection header received
0619 02:49:22.617858 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.617859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.617872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.617925 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.618342 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.619077 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.619847 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.620069 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.620138 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.621320 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.621322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.621322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.621332 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.622675 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.623108 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.623109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.623110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.623285 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.623536 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.623537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.623538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.623911 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.624085 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.624272 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.624402 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:22.646633 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:42593
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:22.751547 29101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:22.751868 29101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:22.751904 29101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:22.753623 29101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:22.753717 29101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:22.753755 29101 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:22.753768 29101 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:22.753789 29101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:22.756049 29101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:22.756434 29101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:22.756755 29101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:22.763796 29109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.763847 29107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:22.763796 29106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:22.763863 29101 server_base.cc:1034] running on GCE node
I20260619 02:49:22.764487 29101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:22.764861 29101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:22.766098 29101 hybrid_clock.cc:648] HybridClock initialized: now 1781837362766037 us; error 85 us; skew 500 ppm
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:22.769660 29101 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:22.770327 29101 webserver.cc:460] Webserver started at http://127.26.98.3:37721/ using document root <none> and password file <none>
I20260619 02:49:22.770613 29101 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:22.770670 29101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:22.770816 29101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:22.772132 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "fef3362cc2cb4102970b32a61a696aa9"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "ed11234e58ed6f4a3e8cd713a4b4f813"
server_key_iv: "2211e5a9e7b3ac0d0f604e340eb548cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.772527 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "fef3362cc2cb4102970b32a61a696aa9"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "ed11234e58ed6f4a3e8cd713a4b4f813"
server_key_iv: "2211e5a9e7b3ac0d0f604e340eb548cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.774292 29101 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:22.775681 29116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.775983 29101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.776111 29101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "fef3362cc2cb4102970b32a61a696aa9"
format_stamp: "Formatted at 2026-06-19 02:49:22 on dist-test-slave-0f9d"
server_key: "ed11234e58ed6f4a3e8cd713a4b4f813"
server_key_iv: "2211e5a9e7b3ac0d0f604e340eb548cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:22.776211 29101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:22.786553 29101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:22.789508 29101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:22.789702 29101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:22.790481 29101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:22.790539 29101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.790573 29101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:22.790588 29101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.800310 29101 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:37251
I20260619 02:49:22.800402 29231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:37251 every 8 connection(s)
I20260619 02:49:22.800809 29101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:22.803654 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29101
I20260619 02:49:22.803771 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:22.803996 27016 external_mini_cluster.cc:1428] Setting key c73b096472c7456014a6fd398e9ed239
I20260619 02:49:22.804333 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.790960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:35649 (local address 127.26.98.62:42593)
0619 02:49:22.791033 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.791036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.791326 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:49:22.792177 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.792178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.792195 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.792270 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.793485 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.794075 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.795530 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.795837 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.795911 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.800639 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.800641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.800642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.800652 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.802215 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.802856 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.802858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.802859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.803057 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.803439 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.803441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.803442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.803634 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.803726 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.804124 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.804227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:49:22.804597 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.790946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.3:35649)
0619 02:49:22.791146 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.791206 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.792042 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.792376 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.792381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.792883 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.793352 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.793363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.794201 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.794202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.795399 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.795405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.795932 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.795934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.796111 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.797405 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.797419 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.800481 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.802337 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.802346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.802349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.802635 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.803134 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.803135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.803135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.803267 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.803750 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.803760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.803952 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.804069 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.804306 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260619 02:49:22.805240 29232 heartbeater.cc:347] Connected to a master server at 127.26.98.62:42593
I20260619 02:49:22.805341 29232 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:22.805578 29232 heartbeater.cc:511] Master 127.26.98.62:42593 requested a full tablet report, sending...
I20260619 02:49:22.806053 28749 ts_manager.cc:194] Registered new tserver with Master: fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251)
I20260619 02:49:22.806708 28749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:35649
I20260619 02:49:22.813732 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.806650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:49141 (local address 127.26.98.62:42593)
0619 02:49:22.806715 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.806717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.806765 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:22.806794 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.806795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.806809 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.806875 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.807373 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.807999 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.808798 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.809035 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.809104 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.810422 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.810424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.810425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.810434 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.812196 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.812693 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.812694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.812694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.812750 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.812928 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.812989 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.812989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.813197 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.813311 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.813504 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.813615 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:22.813774 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.806445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.26.98.3:49141)
0619 02:49:22.806607 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.806621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.806685 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.806951 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.806952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.807208 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.807291 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.807294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.808099 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.808100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.808719 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.808722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.809111 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.809112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.809212 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.809830 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.809835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.810273 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.812385 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.812387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.812387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.812601 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.812807 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.812808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.812808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.812867 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.813330 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.813332 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.813400 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.813466 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.813618 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260619 02:49:22.816831 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.844163 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.835580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.26.98.62:42593)
0619 02:49:22.835687 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.835689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.835701 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:22.835735 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.835736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.835748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.835847 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.836572 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.837122 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.837972 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.838171 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.838234 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.841300 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.841303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.841304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.841314 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.843006 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.843388 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.843389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.843390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.843448 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.843614 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.843614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.843615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.843745 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.843843 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.843959 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.844050 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:49:22.846542 28749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:22.849503 28749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:22.858433 29251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.853007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:41301 (local address 127.0.0.1:58538)
0619 02:49:22.853621 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.853630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.853699 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.853909 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.853910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.853912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:22.854025 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.854027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.855545 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.855546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.857037 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.857044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.857985 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.857986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.858129 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.858147 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.858239 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.858313 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260619 02:49:22.858817 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.853318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.26.98.1:41301)
0619 02:49:22.853543 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.853550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.853705 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:49:22.853771 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.853773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.853791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.854101 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:22.854174 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.855423 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.857200 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.857850 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.857958 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.858046 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.858637 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.858699 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260619 02:49:22.859560 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.852999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44737 (local address 127.0.0.1:47720)
0619 02:49:22.853491 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.853503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.853570 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.854040 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.854041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.854042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:22.854081 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.854082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.857197 (+ 3115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.857198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.857986 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.857989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.859247 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.859249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.859331 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.859338 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.859408 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.859468 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260619 02:49:22.859763 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.853007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37251 (local address 127.0.0.1:52600)
0619 02:49:22.853322 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.853422 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.853522 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.853905 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.853908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.853911 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:22.854003 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.854006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.855333 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.855336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.857038 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.857044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.859434 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.859435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.859524 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.859531 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.859591 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.859685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":69,"thread_start_us":117,"threads_started":1}
I20260619 02:49:22.859810 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.853306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.26.98.3:37251)
0619 02:49:22.853645 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.853650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.853661 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:22.853711 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.853713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.853731 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.853814 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:22.854093 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.855175 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.858106 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.859333 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.859418 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.859478 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.859621 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.859700 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260619 02:49:22.859839 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a04011d8b43aea9604025ef63d32e (DEFAULT_TABLE table=test-table [id=238d423b99c14b5094d3c94b582dda7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:22.860122 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.853118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.26.98.2:44737)
0619 02:49:22.853826 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.853831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.853840 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:22.853912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.853915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.853932 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.854033 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:22.854139 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.855423 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.858185 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.859129 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.859215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.859290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.859956 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.860010 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":109,"threads_started":1}
I20260619 02:49:22.860380 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a04011d8b43aea9604025ef63d32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.860450 29166 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a04011d8b43aea9604025ef63d32e (DEFAULT_TABLE table=test-table [id=238d423b99c14b5094d3c94b582dda7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:22.860841 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a04011d8b43aea9604025ef63d32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.860853 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a04011d8b43aea9604025ef63d32e (DEFAULT_TABLE table=test-table [id=238d423b99c14b5094d3c94b582dda7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:22.861282 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a04011d8b43aea9604025ef63d32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.864267 29255 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Bootstrap starting.
I20260619 02:49:22.865196 29255 tablet_bootstrap.cc:654] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.865170 29256 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Bootstrap starting.
I20260619 02:49:22.865789 29255 log.cc:824] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:22.866333 29256 tablet_bootstrap.cc:654] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.866793 29255 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: No bootstrap required, opened a new log
I20260619 02:49:22.866909 29255 ts_tablet_manager.cc:1397] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:22.867202 29256 log.cc:824] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:22.868054 29256 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: No bootstrap required, opened a new log
I20260619 02:49:22.868132 29256 ts_tablet_manager.cc:1397] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:22.868780 29255 raft_consensus.cc:348] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.868966 29255 raft_consensus.cc:374] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.868990 29255 raft_consensus.cc:729] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e12f461c745428b9db89adbfb0a46, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.869107 29255 consensus_queue.cc:260] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.869454 29255 ts_tablet_manager.cc:1428] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:22.869493 28952 heartbeater.cc:503] Master 127.26.98.62:42593 was elected leader, sending a full tablet report...
I20260619 02:49:22.869760 29258 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Bootstrap starting.
I20260619 02:49:22.870997 29258 tablet_bootstrap.cc:654] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.870783 29256 raft_consensus.cc:348] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.871246 29256 raft_consensus.cc:374] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.871315 29256 raft_consensus.cc:729] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.871402 29258 log.cc:824] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:22.871618 29256 consensus_queue.cc:260] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.872112 29256 ts_tablet_manager.cc:1428] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:22.872146 29232 heartbeater.cc:503] Master 127.26.98.62:42593 was elected leader, sending a full tablet report...
I20260619 02:49:22.872473 29258 tablet_bootstrap.cc:492] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: No bootstrap required, opened a new log
I20260619 02:49:22.872563 29258 ts_tablet_manager.cc:1397] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 02:49:22.874701 29258 raft_consensus.cc:348] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.875018 29258 raft_consensus.cc:374] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.875048 29258 raft_consensus.cc:729] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.875241 29258 consensus_queue.cc:260] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.875636 29258 ts_tablet_manager.cc:1428] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 02:49:22.875747 29092 heartbeater.cc:503] Master 127.26.98.62:42593 was elected leader, sending a full tablet report...
W20260619 02:49:22.926067 28953 tablet.cc:2369] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:22.931864 29263 raft_consensus.cc:482] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:22.932148 29263 raft_consensus.cc:504] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.932667 29263 leader_election.cc:288] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251), 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301)
I20260619 02:49:22.936311 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.933076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:42357 (local address 127.26.98.3:37251)
0619 02:49:22.933131 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.933134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.933148 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:22.933189 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.933190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.933205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.933288 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:22.933479 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.934086 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.935446 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.935896 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.935961 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.935975 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.936174 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.936219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:22.936331 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.933067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:56293 (local address 127.26.98.1:41301)
0619 02:49:22.933135 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.933137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.933387 (+ 250us) server_negotiation.cc:407] Connection header received
0619 02:49:22.933442 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.933443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.933459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.933526 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:22.933711 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.934289 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.935532 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.935950 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.936039 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.936052 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.936184 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.936232 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:22.936342 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.932941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37251 (local address 127.26.98.2:42357)
0619 02:49:22.932985 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.932998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.933048 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.933299 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.933300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.933303 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:22.933400 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.933403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.934201 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.934202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.935343 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.935347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.936000 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.936002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.936091 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.936097 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.936162 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.936218 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 02:49:22.936350 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.932941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:41301 (local address 127.26.98.2:56293)
0619 02:49:22.933304 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.933313 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.933393 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.933538 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.933539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.933541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:22.933612 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.933613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.934382 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.934383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.935414 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.935418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.936046 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.936047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.936113 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.936116 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.936167 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.936211 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":156,"thread_start_us":133,"threads_started":1}
I20260619 02:49:22.936767 29186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a04011d8b43aea9604025ef63d32e" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef3362cc2cb4102970b32a61a696aa9" is_pre_election: true
I20260619 02:49:22.936975 29186 raft_consensus.cc:2454] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 0.
I20260619 02:49:22.937215 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a04011d8b43aea9604025ef63d32e" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46" is_pre_election: true
I20260619 02:49:22.937292 28978 leader_election.cc:302] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be2aaa42aff34448b4ee909dd46b2a54, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:22.937361 28906 raft_consensus.cc:2454] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 0.
I20260619 02:49:22.937469 29263 raft_consensus.cc:2781] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:22.937533 29263 raft_consensus.cc:482] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:22.937557 29263 raft_consensus.cc:3037] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:22.938640 29263 raft_consensus.cc:504] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.938802 29263 leader_election.cc:288] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 election: Requested vote from peers fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251), 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301)
I20260619 02:49:22.939045 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a04011d8b43aea9604025ef63d32e" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46"
I20260619 02:49:22.939143 28906 raft_consensus.cc:3037] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:22.939122 29186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a04011d8b43aea9604025ef63d32e" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef3362cc2cb4102970b32a61a696aa9"
I20260619 02:49:22.939270 29186 raft_consensus.cc:3037] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:22.940008 28906 raft_consensus.cc:2454] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 1.
I20260619 02:49:22.940209 28977 leader_election.cc:302] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, be2aaa42aff34448b4ee909dd46b2a54; no voters:
I20260619 02:49:22.940232 29186 raft_consensus.cc:2454] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 1.
I20260619 02:49:22.940322 29263 raft_consensus.cc:2781] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:22.940483 29263 raft_consensus.cc:686] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 1 LEADER]: Becoming Leader. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Running, Role: LEADER
I20260619 02:49:22.940608 29263 consensus_queue.cc:237] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } }
I20260619 02:49:22.941880 28749 catalog_manager.cc:5495] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "be2aaa42aff34448b4ee909dd46b2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:49:22 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837362, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:22.981644 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.972042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:42593 (local address 127.0.0.1:50986)
0619 02:49:22.972270 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.972282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:22.972418 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:22.972642 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:22.972644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:22.973219 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:22.973311 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.973316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.974150 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.974151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.974747 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:22.974749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.975310 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.975312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:22.975384 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.976035 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:22.976040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:22.977864 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:22.979964 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.979980 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.979983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.980344 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.980654 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:22.980656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:22.980657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:22.980783 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:22.981193 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:22.981207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:22.981299 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 02:49:22.981385 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:49:22.981542 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
I20260619 02:49:22.981741 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.972141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.26.98.62:42593)
0619 02:49:22.972253 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.972255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.972413 (+ 158us) server_negotiation.cc:407] Connection header received
0619 02:49:22.972471 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.972472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.972486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.972570 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.973461 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.974018 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.974926 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.975172 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.975275 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.978057 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.978060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.978061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:22.978071 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.979815 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.980428 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.980429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.980430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.980489 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:22.980906 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:22.980907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:22.980908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:22.981066 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.981150 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.981439 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.981646 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":77}
I20260619 02:49:22.984717 28749 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:50986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:49:22.985001 28749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:22.986771 28886 tablet_service.cc:1467] Processing CreateTablet for tablet a535b89301c342f5b279ff94512542f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dce98bc138af4f349e236eebf811d862]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:22.986848 29166 tablet_service.cc:1467] Processing CreateTablet for tablet a535b89301c342f5b279ff94512542f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dce98bc138af4f349e236eebf811d862]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:22.987048 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a535b89301c342f5b279ff94512542f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.987057 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a535b89301c342f5b279ff94512542f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.988636 29255 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: Bootstrap starting.
I20260619 02:49:22.988844 29026 tablet_service.cc:1467] Processing CreateTablet for tablet a535b89301c342f5b279ff94512542f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dce98bc138af4f349e236eebf811d862]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:22.989050 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a535b89301c342f5b279ff94512542f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:22.989120 29255 tablet_bootstrap.cc:654] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.989471 29256 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: Bootstrap starting.
I20260619 02:49:22.989853 29255 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: No bootstrap required, opened a new log
I20260619 02:49:22.989912 29255 ts_tablet_manager.cc:1397] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.990020 29256 tablet_bootstrap.cc:654] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.990096 29255 raft_consensus.cc:348] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.990214 29255 raft_consensus.cc:374] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.990236 29255 raft_consensus.cc:729] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e12f461c745428b9db89adbfb0a46, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.990288 29255 consensus_queue.cc:260] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.990430 29255 ts_tablet_manager.cc:1428] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:22.990602 29260 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:22.990664 29258 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: Bootstrap starting.
I20260619 02:49:22.990685 29260 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:22.990815 29256 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: No bootstrap required, opened a new log
I20260619 02:49:22.990860 29256 ts_tablet_manager.cc:1397] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.991070 29256 raft_consensus.cc:348] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.991144 29256 raft_consensus.cc:374] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.991163 29256 raft_consensus.cc:729] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.991199 29258 tablet_bootstrap.cc:654] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:22.991210 29256 consensus_queue.cc:260] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.991333 29256 ts_tablet_manager.cc:1428] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:22.991418 29262 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:22.991530 29262 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:22.992004 29258 tablet_bootstrap.cc:492] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: No bootstrap required, opened a new log
I20260619 02:49:22.992061 29258 ts_tablet_manager.cc:1397] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:22.992220 29258 raft_consensus.cc:348] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.992303 29258 raft_consensus.cc:374] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:22.992316 29258 raft_consensus.cc:729] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Initialized, Role: FOLLOWER
I20260619 02:49:22.992367 29258 consensus_queue.cc:260] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:22.992466 29258 ts_tablet_manager.cc:1428] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:22.992596 29263 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:22.992654 29263 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 02:49:23.051712 29233 tablet.cc:2369] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:23.113142 29093 tablet.cc:2369] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:23.229255 29262 raft_consensus.cc:482] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:23.229408 29262 raft_consensus.cc:504] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.229908 29262 leader_election.cc:288] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:23.233314 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.230286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:48313 (local address 127.26.98.2:44737)
0619 02:49:23.230362 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.230364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.230377 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:49:23.230474 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.230476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.230494 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.230564 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:23.230745 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.231333 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.232420 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.232864 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.232970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.232985 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.233163 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.233216 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:49:23.233314 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.230166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44737 (local address 127.26.98.3:48313)
0619 02:49:23.230260 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.230277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.230356 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.230584 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.230586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.230588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:23.230676 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.230679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.231439 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.231441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.232293 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.232297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.232982 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.232984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.233088 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.233092 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.233146 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.233211 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:49:23.233415 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.230464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:43647 (local address 127.26.98.1:41301)
0619 02:49:23.230525 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.230526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.230594 (+ 68us) server_negotiation.cc:407] Connection header received
0619 02:49:23.230697 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.230698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.230711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.230770 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:23.230977 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.231529 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.232558 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.232998 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.233080 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.233093 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.233276 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.233332 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:49:23.233714 29276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.230170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:41301 (local address 127.26.98.3:43647)
0619 02:49:23.230507 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.230516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.230608 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.230778 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.230779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.230781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:23.230899 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.230900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.231634 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.231636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.232444 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.232448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.233115 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.233116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.233200 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.233204 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.233263 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.233330 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":92,"thread_start_us":171,"threads_started":1}
I20260619 02:49:23.233893 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a535b89301c342f5b279ff94512542f9" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54" is_pre_election: true
I20260619 02:49:23.234090 29046 raft_consensus.cc:2454] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:23.234104 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a535b89301c342f5b279ff94512542f9" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46" is_pre_election: true
I20260619 02:49:23.234246 28906 raft_consensus.cc:2454] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:23.234380 29119 leader_election.cc:302] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be2aaa42aff34448b4ee909dd46b2a54, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:23.234589 29262 raft_consensus.cc:2781] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:23.234653 29262 raft_consensus.cc:482] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:23.234680 29262 raft_consensus.cc:3037] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.235695 29262 raft_consensus.cc:504] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.235896 29262 leader_election.cc:288] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:23.236126 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a535b89301c342f5b279ff94512542f9" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46"
I20260619 02:49:23.236234 28906 raft_consensus.cc:3037] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.236356 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a535b89301c342f5b279ff94512542f9" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54"
I20260619 02:49:23.236481 29046 raft_consensus.cc:3037] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.237087 28906 raft_consensus.cc:2454] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:23.237248 29046 raft_consensus.cc:2454] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:23.237342 29117 leader_election.cc:302] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:23.237485 29262 raft_consensus.cc:2781] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:23.237709 29262 raft_consensus.cc:686] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [term 1 LEADER]: Becoming Leader. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Running, Role: LEADER
I20260619 02:49:23.237843 29262 consensus_queue.cc:237] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.238567 29277 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: TxnStatusTablet state changed. Reason: New leader fef3362cc2cb4102970b32a61a696aa9. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.238683 29277 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:23.239194 29279 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:49:23.239348 29279 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:49:23.239451 28749 catalog_manager.cc:5495] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:23.283500 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.280192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.26.98.3:37251)
0619 02:49:23.280290 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.280292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.280303 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:23.280346 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.280347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.280403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.280468 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.280728 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.281343 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.282536 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.282756 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.282829 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.283123 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.283260 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.283355 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.283390 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.283415 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:49:23.283562 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.279824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37251 (local address 127.0.0.1:52610)
0619 02:49:23.279946 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.279964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.280039 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.280502 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.280504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.280505 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:23.280588 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.280594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.281429 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.281430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.282399 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.282403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.282848 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.282849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.282961 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.282997 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.283320 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.283328 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.283382 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.283460 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:49:23.286183 29046 raft_consensus.cc:1261] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.286273 28906 raft_consensus.cc:1261] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.286619 29277 consensus_queue.cc:1035] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:23.286782 29262 consensus_queue.cc:1035] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:23.288636 29263 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: TxnStatusTablet state changed. Reason: New leader fef3362cc2cb4102970b32a61a696aa9. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.288784 29263 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:23.288760 29260 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: TxnStatusTablet state changed. Reason: New leader fef3362cc2cb4102970b32a61a696aa9. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.288865 29260 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:23.290431 29262 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.290561 29262 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:23.291069 29277 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.291157 29277 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P fef3362cc2cb4102970b32a61a696aa9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:23.291986 29263 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.292080 29263 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P be2aaa42aff34448b4ee909dd46b2a54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:23.292153 29260 tablet_replica.cc:439] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } } }
I20260619 02:49:23.292275 29260 tablet_replica.cc:442] T a535b89301c342f5b279ff94512542f9 P 1e9e12f461c745428b9db89adbfb0a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:23.292757 29283 mvcc.cc:205] Tried to move back new op lower bound from 7298405840016519168 to 7298405839822811136. Current Snapshot: MvccSnapshot[applied={T|T < 7298405840016519168 or (T in {7298405840016519168})}]
I20260619 02:49:23.292882 29284 mvcc.cc:205] Tried to move back new op lower bound from 7298405840016519168 to 7298405839822811136. Current Snapshot: MvccSnapshot[applied={T|T < 7298405840016519168}]
I20260619 02:49:23.296727 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.293474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.26.98.2:44737)
0619 02:49:23.293518 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.293521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.293578 (+ 57us) server_negotiation.cc:407] Connection header received
0619 02:49:23.293668 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.293669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.293684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.293744 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.293911 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.294426 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.295689 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.296079 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.296161 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.296368 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.296497 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.296571 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.296614 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.296640 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:49:23.304314 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.300790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37251 (local address 127.26.98.2:33089)
0619 02:49:23.300984 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.300994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.301056 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.301261 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.301263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.301265 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:23.301358 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.301360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.302396 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.302397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.303139 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.303143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.303732 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.303733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.303794 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.303819 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.304091 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.304094 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.304159 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.304221 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260619 02:49:23.304296 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.300934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:33089 (local address 127.26.98.3:37251)
0619 02:49:23.301005 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.301007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.301046 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:49:23.301128 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.301129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.301142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.301210 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.301431 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.302267 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.303223 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.303642 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.303719 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.303932 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.303975 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.304036 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.304171 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.304195 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:49:23.307982 28906 raft_consensus.cc:1261] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Refusing update from remote peer be2aaa42aff34448b4ee909dd46b2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.308085 29186 raft_consensus.cc:1261] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Refusing update from remote peer be2aaa42aff34448b4ee909dd46b2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.308357 29265 consensus_queue.cc:1035] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:23.308530 29263 consensus_queue.cc:1035] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:23.312058 29282 mvcc.cc:205] Tried to move back new op lower bound from 7298405840107257856 to 7298405838605647872. Current Snapshot: MvccSnapshot[applied={T|T < 7298405840107257856}]
I20260619 02:49:23.319547 29252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.316352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:53851 (local address 127.26.98.2:44737)
0619 02:49:23.316413 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.316415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.316427 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:23.316455 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.316456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.316469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.316513 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.316663 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.317357 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.318370 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.318648 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.318744 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.319239 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.319278 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.319326 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.319404 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.319435 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:49:23.319586 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.316121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44737 (local address 127.26.98.3:53851)
0619 02:49:23.316325 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.316339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.316405 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.316527 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.316527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.316530 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:23.316603 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.316605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.317467 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.317468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.318295 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.318300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.318829 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.318832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.319029 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.319057 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.319354 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.319356 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.319400 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.319475 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260619 02:49:23.332636 28748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:23.332837 28748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:23.334177 28748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Sending DeleteTablet for 3 replicas of tablet 5e1a04011d8b43aea9604025ef63d32e
I20260619 02:49:23.334986 29166 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a04011d8b43aea9604025ef63d32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:23 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260619 02:49:23.335346 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a04011d8b43aea9604025ef63d32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:23 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260619 02:49:23.335719 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a04011d8b43aea9604025ef63d32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:23 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47720
I20260619 02:49:23.335690 28748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:23.335919 28748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:23.336263 29303 tablet_replica.cc:330] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: stopping tablet replica
I20260619 02:49:23.336459 29303 raft_consensus.cc:2229] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:23.336539 29303 raft_consensus.cc:2258] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:23.336983 29304 tablet_replica.cc:330] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: stopping tablet replica
I20260619 02:49:23.337141 29304 raft_consensus.cc:2229] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:23.337219 29304 raft_consensus.cc:2258] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:23.337422 29305 tablet_replica.cc:330] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: stopping tablet replica
I20260619 02:49:23.337544 29305 raft_consensus.cc:2229] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:23.338255 28886 tablet_service.cc:1467] Processing CreateTablet for tablet c86e72be79f3406a92f7db995158c983 (DEFAULT_TABLE table=test-table [id=d52b2df10ca24f96a5e313113e12dcb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:23.338474 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86e72be79f3406a92f7db995158c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:23.340852 29166 tablet_service.cc:1467] Processing CreateTablet for tablet c86e72be79f3406a92f7db995158c983 (DEFAULT_TABLE table=test-table [id=d52b2df10ca24f96a5e313113e12dcb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:23.341066 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86e72be79f3406a92f7db995158c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:23.342957 29256 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Bootstrap starting.
I20260619 02:49:23.343143 29255 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Bootstrap starting.
I20260619 02:49:23.343578 29256 tablet_bootstrap.cc:654] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:23.343683 29255 tablet_bootstrap.cc:654] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:23.344362 29026 tablet_service.cc:1467] Processing CreateTablet for tablet c86e72be79f3406a92f7db995158c983 (DEFAULT_TABLE table=test-table [id=d52b2df10ca24f96a5e313113e12dcb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:23.344998 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86e72be79f3406a92f7db995158c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:23.345626 29256 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: No bootstrap required, opened a new log
I20260619 02:49:23.345711 29256 ts_tablet_manager.cc:1397] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:23.346053 29256 raft_consensus.cc:348] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.346185 29256 raft_consensus.cc:374] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:23.346206 29256 raft_consensus.cc:729] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Initialized, Role: FOLLOWER
I20260619 02:49:23.346282 29256 consensus_queue.cc:260] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.346403 29256 ts_tablet_manager.cc:1428] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:23.346797 29258 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Bootstrap starting.
I20260619 02:49:23.347481 29303 ts_tablet_manager.cc:1905] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:23.347656 29258 tablet_bootstrap.cc:654] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:23.337806 29305 raft_consensus.cc:2258] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:23.348991 29305 ts_tablet_manager.cc:1905] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:23.349540 29303 ts_tablet_manager.cc:1918] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:23.349629 29303 log.cc:1196] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/5e1a04011d8b43aea9604025ef63d32e
I20260619 02:49:23.349922 29303 ts_tablet_manager.cc:1939] T 5e1a04011d8b43aea9604025ef63d32e P fef3362cc2cb4102970b32a61a696aa9: Deleting consensus metadata
I20260619 02:49:23.350546 28734 catalog_manager.cc:4841] TS fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251): tablet 5e1a04011d8b43aea9604025ef63d32e (table test-table [id=238d423b99c14b5094d3c94b582dda7a]) successfully deleted
I20260619 02:49:23.351086 29304 ts_tablet_manager.cc:1905] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:23.353178 29304 ts_tablet_manager.cc:1918] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:23.353289 29304 log.cc:1196] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/5e1a04011d8b43aea9604025ef63d32e
I20260619 02:49:23.353606 29304 ts_tablet_manager.cc:1939] T 5e1a04011d8b43aea9604025ef63d32e P 1e9e12f461c745428b9db89adbfb0a46: Deleting consensus metadata
I20260619 02:49:23.354338 28733 catalog_manager.cc:4841] TS 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301): tablet 5e1a04011d8b43aea9604025ef63d32e (table test-table [id=238d423b99c14b5094d3c94b582dda7a]) successfully deleted
I20260619 02:49:23.355352 29305 ts_tablet_manager.cc:1918] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:23.355412 29255 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: No bootstrap required, opened a new log
I20260619 02:49:23.355458 29305 log.cc:1196] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/5e1a04011d8b43aea9604025ef63d32e
I20260619 02:49:23.355469 29255 ts_tablet_manager.cc:1397] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260619 02:49:23.355667 29255 raft_consensus.cc:348] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.355757 29255 raft_consensus.cc:374] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:23.355803 29255 raft_consensus.cc:729] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e12f461c745428b9db89adbfb0a46, State: Initialized, Role: FOLLOWER
I20260619 02:49:23.355824 29305 ts_tablet_manager.cc:1939] T 5e1a04011d8b43aea9604025ef63d32e P be2aaa42aff34448b4ee909dd46b2a54: Deleting consensus metadata
I20260619 02:49:23.355860 29255 consensus_queue.cc:260] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.355957 29255 ts_tablet_manager.cc:1428] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:23.356841 28735 catalog_manager.cc:4841] TS be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737): tablet 5e1a04011d8b43aea9604025ef63d32e (table test-table [id=238d423b99c14b5094d3c94b582dda7a]) successfully deleted
I20260619 02:49:23.357509 29258 tablet_bootstrap.cc:492] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: No bootstrap required, opened a new log
I20260619 02:49:23.357581 29258 ts_tablet_manager.cc:1397] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260619 02:49:23.357800 29258 raft_consensus.cc:348] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.357905 29258 raft_consensus.cc:374] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:23.357928 29258 raft_consensus.cc:729] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Initialized, Role: FOLLOWER
I20260619 02:49:23.357986 29258 consensus_queue.cc:260] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.358109 29258 ts_tablet_manager.cc:1428] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:23.360412 29277 raft_consensus.cc:482] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:23.360538 29277 raft_consensus.cc:504] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.360754 29277 leader_election.cc:288] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:23.361080 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86e72be79f3406a92f7db995158c983" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46" is_pre_election: true
I20260619 02:49:23.361140 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86e72be79f3406a92f7db995158c983" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54" is_pre_election: true
I20260619 02:49:23.361207 28906 raft_consensus.cc:2454] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:23.361243 29046 raft_consensus.cc:2454] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:23.361392 29119 leader_election.cc:302] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be2aaa42aff34448b4ee909dd46b2a54, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:23.361531 29277 raft_consensus.cc:2781] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:23.361567 29277 raft_consensus.cc:482] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:23.361588 29277 raft_consensus.cc:3037] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.362784 29277 raft_consensus.cc:504] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.363054 29277 leader_election.cc:288] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:23.363358 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86e72be79f3406a92f7db995158c983" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54"
I20260619 02:49:23.363377 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86e72be79f3406a92f7db995158c983" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46"
I20260619 02:49:23.363467 29046 raft_consensus.cc:3037] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.363474 28906 raft_consensus.cc:3037] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:23.364435 29046 raft_consensus.cc:2454] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:23.364452 28906 raft_consensus.cc:2454] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:23.364675 29117 leader_election.cc:302] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:23.364820 29277 raft_consensus.cc:2781] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:23.364871 29277 raft_consensus.cc:686] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 1 LEADER]: Becoming Leader. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Running, Role: LEADER
I20260619 02:49:23.364921 29277 consensus_queue.cc:237] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:23.365621 28746 catalog_manager.cc:5495] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:23.381304 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.377966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.26.98.3:37251)
0619 02:49:23.378017 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.378019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.378026 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:23.378056 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.378057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.378070 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.378113 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.378308 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.379113 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.380290 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.380597 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.380657 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.380877 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.380921 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.380961 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.381148 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.381170 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:23.388967 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.385001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:53349 (local address 127.26.98.3:37251)
0619 02:49:23.385041 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:23.385043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:23.385052 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:23.385132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:23.385133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:23.385147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:23.385215 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:23.385395 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.386136 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.387712 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.388006 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.388362 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.388555 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.388615 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.388679 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:23.388801 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:49:23.388834 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:23.389066 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:23.384906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37251 (local address 127.26.98.3:53349)
0619 02:49:23.384979 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:23.384996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:23.385060 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:23.385234 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:23.385235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:23.385237 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:23.385316 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.385319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.386284 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.386285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.387470 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:23.387478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:23.388363 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:23.388365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:23.388455 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:23.388466 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:23.388716 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:23.388719 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:23.388791 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:49:23.388866 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:49:23.392313 29046 raft_consensus.cc:1261] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.392313 28906 raft_consensus.cc:1261] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:23.392601 29277 consensus_queue.cc:1035] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:23.392782 29262 consensus_queue.cc:1035] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:49:23.401312 29282 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:49:24.927412 28748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:24.927575 28748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:24.928798 28748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Sending DeleteTablet for 3 replicas of tablet c86e72be79f3406a92f7db995158c983
I20260619 02:49:24.929342 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet c86e72be79f3406a92f7db995158c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:24 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47720
I20260619 02:49:24.929342 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet c86e72be79f3406a92f7db995158c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:24 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260619 02:49:24.929630 29166 tablet_service.cc:1514] Processing DeleteTablet for tablet c86e72be79f3406a92f7db995158c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:24 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260619 02:49:24.929885 29334 tablet_replica.cc:330] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: stopping tablet replica
I20260619 02:49:24.929978 29334 raft_consensus.cc:2229] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:24.930052 29334 raft_consensus.cc:2258] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:24.930385 29333 tablet_replica.cc:330] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: stopping tablet replica
I20260619 02:49:24.930457 29333 raft_consensus.cc:2229] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:24.930505 29333 raft_consensus.cc:2258] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:24.930980 29334 ts_tablet_manager.cc:1905] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:24.932986 29334 ts_tablet_manager.cc:1918] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:24.933082 29334 log.cc:1196] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/c86e72be79f3406a92f7db995158c983
I20260619 02:49:24.933306 29334 ts_tablet_manager.cc:1939] T c86e72be79f3406a92f7db995158c983 P be2aaa42aff34448b4ee909dd46b2a54: Deleting consensus metadata
I20260619 02:49:24.933815 28735 catalog_manager.cc:4841] TS be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737): tablet c86e72be79f3406a92f7db995158c983 (table test-table [id=d52b2df10ca24f96a5e313113e12dcb5]) successfully deleted
I20260619 02:49:24.933851 29333 ts_tablet_manager.cc:1905] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:24.935083 29335 tablet_replica.cc:330] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: stopping tablet replica
I20260619 02:49:24.935204 29335 raft_consensus.cc:2229] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:24.935353 29335 raft_consensus.cc:2258] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:24.935719 29335 ts_tablet_manager.cc:1905] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:24.935715 29333 ts_tablet_manager.cc:1918] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:24.935796 29333 log.cc:1196] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/c86e72be79f3406a92f7db995158c983
I20260619 02:49:24.936043 29333 ts_tablet_manager.cc:1939] T c86e72be79f3406a92f7db995158c983 P 1e9e12f461c745428b9db89adbfb0a46: Deleting consensus metadata
I20260619 02:49:24.936522 28733 catalog_manager.cc:4841] TS 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301): tablet c86e72be79f3406a92f7db995158c983 (table test-table [id=d52b2df10ca24f96a5e313113e12dcb5]) successfully deleted
I20260619 02:49:24.937585 29335 ts_tablet_manager.cc:1918] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:24.937666 29335 log.cc:1196] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/c86e72be79f3406a92f7db995158c983
I20260619 02:49:24.937944 29335 ts_tablet_manager.cc:1939] T c86e72be79f3406a92f7db995158c983 P fef3362cc2cb4102970b32a61a696aa9: Deleting consensus metadata
I20260619 02:49:24.938393 28734 catalog_manager.cc:4841] TS fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251): tablet c86e72be79f3406a92f7db995158c983 (table test-table [id=d52b2df10ca24f96a5e313113e12dcb5]) successfully deleted
Jun 19 02:49:24 dist-test-slave-0f9d krb5kdc[28704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:24 dist-test-slave-0f9d krb5kdc[28704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:24.967569 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:24.953084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.26.98.62:42593)
0619 02:49:24.953247 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:24.953249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:24.953260 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:24.953360 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:24.953361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:24.953376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:24.953442 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:24.954065 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.954635 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.955587 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.955845 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.955906 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:24.964248 (+ 8342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:24.964251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:24.964252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:24.964265 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:24.965977 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.966539 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.966541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.966541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.966637 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.966871 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.966872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.966872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.967080 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:24.967177 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:24.967309 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:49:24.967475 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260619 02:49:24.970053 28748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:24.970309 28748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:24.972630 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 714ddfd7e644401884796f50eaa0a275 (DEFAULT_TABLE table=test-table [id=289f1df9a01544399ec5c74e7167c24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:24.972757 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 714ddfd7e644401884796f50eaa0a275 (DEFAULT_TABLE table=test-table [id=289f1df9a01544399ec5c74e7167c24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:24.972783 29166 tablet_service.cc:1467] Processing CreateTablet for tablet 714ddfd7e644401884796f50eaa0a275 (DEFAULT_TABLE table=test-table [id=289f1df9a01544399ec5c74e7167c24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:24.972842 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714ddfd7e644401884796f50eaa0a275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:24.972985 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714ddfd7e644401884796f50eaa0a275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:24.972982 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714ddfd7e644401884796f50eaa0a275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:24.974387 29344 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Bootstrap starting.
I20260619 02:49:24.974983 29344 tablet_bootstrap.cc:654] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:24.975386 29347 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Bootstrap starting.
I20260619 02:49:24.975549 29346 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Bootstrap starting.
I20260619 02:49:24.975780 29347 tablet_bootstrap.cc:654] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:24.975773 29344 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: No bootstrap required, opened a new log
I20260619 02:49:24.975832 29344 ts_tablet_manager.cc:1397] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:24.976027 29346 tablet_bootstrap.cc:654] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:24.976002 29344 raft_consensus.cc:348] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.976101 29344 raft_consensus.cc:374] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:24.976133 29344 raft_consensus.cc:729] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Initialized, Role: FOLLOWER
I20260619 02:49:24.976184 29344 consensus_queue.cc:260] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.976356 29347 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: No bootstrap required, opened a new log
I20260619 02:49:24.976389 29344 ts_tablet_manager.cc:1428] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:24.976408 29347 ts_tablet_manager.cc:1397] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:24.976589 29347 raft_consensus.cc:348] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.976665 29347 raft_consensus.cc:374] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:24.976684 29347 raft_consensus.cc:729] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e12f461c745428b9db89adbfb0a46, State: Initialized, Role: FOLLOWER
I20260619 02:49:24.976732 29347 consensus_queue.cc:260] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.976904 29347 ts_tablet_manager.cc:1428] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:24.977357 29346 tablet_bootstrap.cc:492] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: No bootstrap required, opened a new log
I20260619 02:49:24.977497 29346 ts_tablet_manager.cc:1397] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:24.977658 29346 raft_consensus.cc:348] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.977741 29346 raft_consensus.cc:374] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:24.977771 29346 raft_consensus.cc:729] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Initialized, Role: FOLLOWER
I20260619 02:49:24.977823 29346 consensus_queue.cc:260] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:24.978010 29346 ts_tablet_manager.cc:1428] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:25.051584 29321 raft_consensus.cc:482] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:25.052019 29321 raft_consensus.cc:504] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.052250 29321 leader_election.cc:288] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:25.052486 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714ddfd7e644401884796f50eaa0a275" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54" is_pre_election: true
I20260619 02:49:25.052486 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714ddfd7e644401884796f50eaa0a275" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46" is_pre_election: true
I20260619 02:49:25.052603 28906 raft_consensus.cc:2454] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:25.052601 29046 raft_consensus.cc:2454] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 0.
I20260619 02:49:25.052769 29119 leader_election.cc:302] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be2aaa42aff34448b4ee909dd46b2a54, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:25.052965 29321 raft_consensus.cc:2781] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:25.053010 29321 raft_consensus.cc:482] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:25.053023 29321 raft_consensus.cc:3037] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.053902 29321 raft_consensus.cc:504] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.054032 29321 leader_election.cc:288] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737)
I20260619 02:49:25.054237 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714ddfd7e644401884796f50eaa0a275" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2aaa42aff34448b4ee909dd46b2a54"
I20260619 02:49:25.054250 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714ddfd7e644401884796f50eaa0a275" candidate_uuid: "fef3362cc2cb4102970b32a61a696aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46"
I20260619 02:49:25.054332 28906 raft_consensus.cc:3037] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.054332 29046 raft_consensus.cc:3037] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.055069 28906 raft_consensus.cc:2454] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:25.055248 29117 leader_election.cc:302] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, fef3362cc2cb4102970b32a61a696aa9; no voters:
I20260619 02:49:25.055303 29046 raft_consensus.cc:2454] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef3362cc2cb4102970b32a61a696aa9 in term 1.
I20260619 02:49:25.055346 29321 raft_consensus.cc:2781] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:25.055413 29321 raft_consensus.cc:686] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 1 LEADER]: Becoming Leader. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Running, Role: LEADER
I20260619 02:49:25.055482 29321 consensus_queue.cc:237] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.056288 28746 catalog_manager.cc:5495] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "fef3362cc2cb4102970b32a61a696aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:25.079339 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:25.076509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.26.98.3:37251)
0619 02:49:25.076735 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:25.076737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:25.076745 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:25.076800 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:25.076801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:25.076816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:25.076897 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:25.077045 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:25.077586 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:25.078519 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:25.078746 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:25.078814 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:25.079063 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:25.079096 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:25.079135 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:25.079178 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 02:49:25.079228 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260619 02:49:25.082086 28906 raft_consensus.cc:1261] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:25.082124 29046 raft_consensus.cc:1261] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Refusing update from remote peer fef3362cc2cb4102970b32a61a696aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:25.082324 29262 consensus_queue.cc:1035] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:25.082499 29262 consensus_queue.cc:1035] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:25.093633 28746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:25.093783 28746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:25.094507 28746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Sending DeleteTablet for 3 replicas of tablet 714ddfd7e644401884796f50eaa0a275
I20260619 02:49:25.094766 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ddfd7e644401884796f50eaa0a275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:25 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260619 02:49:25.094789 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ddfd7e644401884796f50eaa0a275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:25 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47720
I20260619 02:49:25.094923 29333 tablet_replica.cc:330] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: stopping tablet replica
I20260619 02:49:25.094954 29334 tablet_replica.cc:330] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: stopping tablet replica
I20260619 02:49:25.094929 29166 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ddfd7e644401884796f50eaa0a275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:25 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260619 02:49:25.094995 29333 raft_consensus.cc:2229] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:25.095017 29334 raft_consensus.cc:2229] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:25.095055 29333 raft_consensus.cc:2258] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:25.095062 29334 raft_consensus.cc:2258] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:25.095141 29335 tablet_replica.cc:330] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: stopping tablet replica
I20260619 02:49:25.095207 29335 raft_consensus.cc:2229] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:25.095292 29335 raft_consensus.cc:2258] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:25.095619 29335 ts_tablet_manager.cc:1905] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:25.095724 29333 ts_tablet_manager.cc:1905] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:25.095878 29334 ts_tablet_manager.cc:1905] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:25.095912 28746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:25.096083 28746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:25.098362 29335 ts_tablet_manager.cc:1918] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:25.098362 29333 ts_tablet_manager.cc:1918] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:25.098443 29335 log.cc:1196] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/714ddfd7e644401884796f50eaa0a275
I20260619 02:49:25.098443 29333 log.cc:1196] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/714ddfd7e644401884796f50eaa0a275
I20260619 02:49:25.098665 29333 ts_tablet_manager.cc:1939] T 714ddfd7e644401884796f50eaa0a275 P 1e9e12f461c745428b9db89adbfb0a46: Deleting consensus metadata
I20260619 02:49:25.098690 29335 ts_tablet_manager.cc:1939] T 714ddfd7e644401884796f50eaa0a275 P fef3362cc2cb4102970b32a61a696aa9: Deleting consensus metadata
I20260619 02:49:25.098702 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 (DEFAULT_TABLE table=test-table [id=ffb8c5a558d84f818a74ee6a44f721b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:25.098726 29334 ts_tablet_manager.cc:1918] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:25.098780 29334 log.cc:1196] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/714ddfd7e644401884796f50eaa0a275
I20260619 02:49:25.098820 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:25.099056 29166 tablet_service.cc:1467] Processing CreateTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 (DEFAULT_TABLE table=test-table [id=ffb8c5a558d84f818a74ee6a44f721b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:25.099094 29334 ts_tablet_manager.cc:1939] T 714ddfd7e644401884796f50eaa0a275 P be2aaa42aff34448b4ee909dd46b2a54: Deleting consensus metadata
I20260619 02:49:25.099187 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:25.099315 28734 catalog_manager.cc:4841] TS fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251): tablet 714ddfd7e644401884796f50eaa0a275 (table test-table [id=289f1df9a01544399ec5c74e7167c24b]) successfully deleted
I20260619 02:49:25.099658 28733 catalog_manager.cc:4841] TS 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301): tablet 714ddfd7e644401884796f50eaa0a275 (table test-table [id=289f1df9a01544399ec5c74e7167c24b]) successfully deleted
I20260619 02:49:25.100315 29347 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46: Bootstrap starting.
I20260619 02:49:25.100379 28735 catalog_manager.cc:4841] TS be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737): tablet 714ddfd7e644401884796f50eaa0a275 (table test-table [id=289f1df9a01544399ec5c74e7167c24b]) successfully deleted
I20260619 02:49:25.100589 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 (DEFAULT_TABLE table=test-table [id=ffb8c5a558d84f818a74ee6a44f721b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:25.100710 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:25.100775 29346 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Bootstrap starting.
I20260619 02:49:25.101133 29346 tablet_bootstrap.cc:654] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:25.101137 29347 tablet_bootstrap.cc:654] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:25.101787 29346 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: No bootstrap required, opened a new log
I20260619 02:49:25.101846 29346 ts_tablet_manager.cc:1397] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:25.101980 29344 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Bootstrap starting.
I20260619 02:49:25.102005 29346 raft_consensus.cc:348] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.102085 29346 raft_consensus.cc:374] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:25.102104 29346 raft_consensus.cc:729] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef3362cc2cb4102970b32a61a696aa9, State: Initialized, Role: FOLLOWER
I20260619 02:49:25.102173 29346 consensus_queue.cc:260] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.102303 29346 ts_tablet_manager.cc:1428] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:25.102416 29344 tablet_bootstrap.cc:654] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:25.102557 29347 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46: No bootstrap required, opened a new log
I20260619 02:49:25.102609 29347 ts_tablet_manager.cc:1397] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:25.102784 29347 raft_consensus.cc:348] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.102862 29347 raft_consensus.cc:374] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:25.102926 29347 raft_consensus.cc:729] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9e12f461c745428b9db89adbfb0a46, State: Initialized, Role: FOLLOWER
I20260619 02:49:25.102973 29347 consensus_queue.cc:260] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.103070 29347 ts_tablet_manager.cc:1428] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:25.103228 29344 tablet_bootstrap.cc:492] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: No bootstrap required, opened a new log
I20260619 02:49:25.103276 29344 ts_tablet_manager.cc:1397] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:25.103394 29344 raft_consensus.cc:348] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.103448 29344 raft_consensus.cc:374] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:25.103466 29344 raft_consensus.cc:729] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Initialized, Role: FOLLOWER
I20260619 02:49:25.103515 29344 consensus_queue.cc:260] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.103611 29344 ts_tablet_manager.cc:1428] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:25.260264 29350 raft_consensus.cc:482] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:25.260416 29350 raft_consensus.cc:504] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.260610 29350 leader_election.cc:288] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251)
I20260619 02:49:25.261097 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660cc3c8e4bb4b5fb9a8ca58da5cacf7" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46" is_pre_election: true
I20260619 02:49:25.261088 29186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660cc3c8e4bb4b5fb9a8ca58da5cacf7" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef3362cc2cb4102970b32a61a696aa9" is_pre_election: true
I20260619 02:49:25.261258 28906 raft_consensus.cc:2454] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 0.
I20260619 02:49:25.261255 29186 raft_consensus.cc:2454] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 0.
I20260619 02:49:25.261466 28977 leader_election.cc:302] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, be2aaa42aff34448b4ee909dd46b2a54; no voters:
I20260619 02:49:25.261595 29350 raft_consensus.cc:2781] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:25.261638 29350 raft_consensus.cc:482] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:25.261667 29350 raft_consensus.cc:3037] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.262538 29350 raft_consensus.cc:504] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.262688 29350 leader_election.cc:288] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 election: Requested vote from peers 1e9e12f461c745428b9db89adbfb0a46 (127.26.98.1:41301), fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251)
I20260619 02:49:25.262945 28906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660cc3c8e4bb4b5fb9a8ca58da5cacf7" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9e12f461c745428b9db89adbfb0a46"
I20260619 02:49:25.263044 28906 raft_consensus.cc:3037] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.263015 29186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660cc3c8e4bb4b5fb9a8ca58da5cacf7" candidate_uuid: "be2aaa42aff34448b4ee909dd46b2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef3362cc2cb4102970b32a61a696aa9"
I20260619 02:49:25.263118 29186 raft_consensus.cc:3037] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:25.263806 29186 raft_consensus.cc:2454] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 1.
I20260619 02:49:25.263808 28906 raft_consensus.cc:2454] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be2aaa42aff34448b4ee909dd46b2a54 in term 1.
I20260619 02:49:25.264029 28977 leader_election.cc:302] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9e12f461c745428b9db89adbfb0a46, be2aaa42aff34448b4ee909dd46b2a54; no voters:
I20260619 02:49:25.264171 29350 raft_consensus.cc:2781] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:25.264323 29350 raft_consensus.cc:686] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 LEADER]: Becoming Leader. State: Replica: be2aaa42aff34448b4ee909dd46b2a54, State: Running, Role: LEADER
I20260619 02:49:25.264379 29350 consensus_queue.cc:237] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } }
I20260619 02:49:25.265221 28746 catalog_manager.cc:5495] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "be2aaa42aff34448b4ee909dd46b2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be2aaa42aff34448b4ee909dd46b2a54" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:25.295410 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:25.292111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.26.98.2:44737)
0619 02:49:25.292390 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:25.292392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:25.292400 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:25.292424 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:25.292424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:25.292439 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:25.292515 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:25.292835 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:25.293412 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:25.294333 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:25.294575 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:25.294659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:25.294913 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:25.294965 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:25.295015 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:25.295240 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:49:25.295298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":165,"threads_started":1}
I20260619 02:49:25.299010 28906 raft_consensus.cc:1261] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P 1e9e12f461c745428b9db89adbfb0a46 [term 1 FOLLOWER]: Refusing update from remote peer be2aaa42aff34448b4ee909dd46b2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:25.299031 29186 raft_consensus.cc:1261] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Refusing update from remote peer be2aaa42aff34448b4ee909dd46b2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:25.299340 29362 consensus_queue.cc:1035] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef3362cc2cb4102970b32a61a696aa9" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:25.299496 29350 consensus_queue.cc:1035] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9e12f461c745428b9db89adbfb0a46" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:49:25.305328 29317 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:49:26.830235 28746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:26.830386 28746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:26.831686 28746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61b9af13c224a4a9253908418bbf0a4: Sending DeleteTablet for 3 replicas of tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7
I20260619 02:49:26.832248 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:26 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47720
I20260619 02:49:26.832321 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:26 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260619 02:49:26.832453 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28821
I20260619 02:49:26.832557 29166 tablet_service.cc:1514] Processing DeleteTablet for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:26 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260619 02:49:26.833683 29396 tablet_replica.cc:330] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: stopping tablet replica
I20260619 02:49:26.833770 29396 raft_consensus.cc:2229] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:26.833812 29396 raft_consensus.cc:2258] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:26.834218 29396 ts_tablet_manager.cc:1905] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:26.836344 29394 tablet_replica.cc:330] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: stopping tablet replica
I20260619 02:49:26.836459 29394 raft_consensus.cc:2229] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:26.836560 29394 raft_consensus.cc:2258] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:26.836585 29396 ts_tablet_manager.cc:1918] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:26.836722 29396 log.cc:1196] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/660cc3c8e4bb4b5fb9a8ca58da5cacf7
I20260619 02:49:26.836846 29394 ts_tablet_manager.cc:1905] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:26.836992 29396 ts_tablet_manager.cc:1939] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P fef3362cc2cb4102970b32a61a696aa9: Deleting consensus metadata
I20260619 02:49:26.837561 28734 catalog_manager.cc:4841] TS fef3362cc2cb4102970b32a61a696aa9 (127.26.98.3:37251): tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7 (table test-table [id=ffb8c5a558d84f818a74ee6a44f721b0]) successfully deleted
W20260619 02:49:26.839244 28733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:41301 (error 108)
I20260619 02:49:26.839355 29394 ts_tablet_manager.cc:1918] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:26.839429 29394 log.cc:1196] T 660cc3c8e4bb4b5fb9a8ca58da5cacf7 P be2aaa42aff34448b4ee909dd46b2a54: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/660cc3c8e4bb4b5fb9a8ca58da5cacf7
I20260619 02:49:26.839637 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28961
I20260619 02:49:26.846117 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29101
I20260619 02:49:26.847059 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:26.845907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44737 (local address 127.0.0.1:59002)
0619 02:49:26.846128 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:26.846980 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:44737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":1280,"thread_start_us":137,"threads_started":1}
W20260619 02:49:26.847249 28735 catalog_manager.cc:4584] TS be2aaa42aff34448b4ee909dd46b2a54 (127.26.98.2:44737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 660cc3c8e4bb4b5fb9a8ca58da5cacf7: Network error: Client connection negotiation failed: client connection to 127.26.98.2:44737: connect: Connection refused (error 111)
I20260619 02:49:26.848148 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:26.845907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:41301 (local address 127.0.0.1:34234)
0619 02:49:26.846128 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:26.846195 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:41301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":4,"thread_start_us":133,"threads_started":1}
I20260619 02:49:26.852267 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 28717
2026-06-19T02:49:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5961 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:26 dist-test-slave-0f9d krb5kdc[29402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:26 dist-test-slave-0f9d krb5kdc[29402](info): set up 2 sockets
Jun 19 02:49:26 dist-test-slave-0f9d krb5kdc[29402](info): commencing operation
krb5kdc: starting...
W20260619 02:49:27.855203 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:27 dist-test-slave-0f9d krb5kdc[29402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:27.991474 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34041
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:34041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:28.091295 29415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:28.091603 29415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:28.091634 29415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:28.093182 29415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:28.093276 29415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:28.093292 29415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:28.093308 29415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:28.093321 29415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:28.095440 29415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:34041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:28.095785 29415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:28.096036 29415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:28.102209 29423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.102185 29421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.102249 29420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:28.102768 29415 server_base.cc:1034] running on GCE node
I20260619 02:49:28.103119 29415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:28.103515 29415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:28.104846 29415 hybrid_clock.cc:648] HybridClock initialized: now 1781837368104812 us; error 43 us; skew 500 ppm
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:28.109179 29415 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:28.110577 29415 webserver.cc:460] Webserver started at http://127.26.98.62:38815/ using document root <none> and password file <none>
I20260619 02:49:28.110936 29415 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:28.111011 29415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:28.111184 29415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:28.112783 29415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "02700744187b4aa9a3a37c35ee4bf72c"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "485dbbad9cb63229ea87959cdf3af45e"
server_key_iv: "0ec901f465ab967ce33d470781c775c2"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.113461 29415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "02700744187b4aa9a3a37c35ee4bf72c"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "485dbbad9cb63229ea87959cdf3af45e"
server_key_iv: "0ec901f465ab967ce33d470781c775c2"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.115381 29415 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:49:28.116832 29430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.117312 29415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:28.117441 29415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "02700744187b4aa9a3a37c35ee4bf72c"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "485dbbad9cb63229ea87959cdf3af45e"
server_key_iv: "0ec901f465ab967ce33d470781c775c2"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.117560 29415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:28.132416 29415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:28.135834 29415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:28.136026 29415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:28.142397 29415 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:34041
I20260619 02:49:28.142453 29482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:34041 every 8 connection(s)
I20260619 02:49:28.143172 29415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:28.144317 29483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:28.147593 29483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: Bootstrap starting.
I20260619 02:49:28.148610 29483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:28.148682 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29415
I20260619 02:49:28.148762 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:28.148968 27016 external_mini_cluster.cc:1428] Setting key 62779187b69c1803c0adbfb6f510de74
I20260619 02:49:28.149415 29483 log.cc:824] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:28.150707 29483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: No bootstrap required, opened a new log
I20260619 02:49:28.154376 29483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } }
I20260619 02:49:28.155580 29483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:28.156085 29483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02700744187b4aa9a3a37c35ee4bf72c, State: Initialized, Role: FOLLOWER
I20260619 02:49:28.156422 29483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } }
I20260619 02:49:28.156665 29483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:28.156919 29483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:28.157032 29483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:28.158556 29483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } }
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:28.158725 29483 leader_election.cc:302] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02700744187b4aa9a3a37c35ee4bf72c; no voters:
I20260619 02:49:28.159112 29483 leader_election.cc:288] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:28.159149 29488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:28.159317 29488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [term 1 LEADER]: Becoming Leader. State: Replica: 02700744187b4aa9a3a37c35ee4bf72c, State: Running, Role: LEADER
I20260619 02:49:28.159490 29483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:28.159513 29488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } }
I20260619 02:49:28.160460 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02700744187b4aa9a3a37c35ee4bf72c. Latest consensus state: current_term: 1 leader_uuid: "02700744187b4aa9a3a37c35ee4bf72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } } }
I20260619 02:49:28.160614 29490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02700744187b4aa9a3a37c35ee4bf72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02700744187b4aa9a3a37c35ee4bf72c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34041 } } }
I20260619 02:49:28.161126 29490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:28.161249 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:28.161806 29502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:28.161895 29502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:28.161985 29504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:28.162204 29504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:28.163641 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.149564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.26.98.62:34041)
0619 02:49:28.149828 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.149838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.149875 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:49:28.150577 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.150588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.150660 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.150914 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.152410 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.153514 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.155659 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.156247 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.156401 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.159359 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.159367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.159367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:28.159387 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.161750 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.162273 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.162278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.162278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.162394 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.162612 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.162613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.162614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.162958 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.163151 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.163261 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.163438 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":116,"threads_started":1}
I20260619 02:49:28.164702 29504 catalog_manager.cc:1347] Generated new cluster ID: 9096c3e425b34dc88d41b70ca96fd3ab
I20260619 02:49:28.164773 29504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:28.180277 29504 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:28.181209 29504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:28.186321 29504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02700744187b4aa9a3a37c35ee4bf72c: Generated new TSK 0
I20260619 02:49:28.186595 29504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:28.237136 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:28.359833 29509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:28.360128 29509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:28.360169 29509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:28.361824 29509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:28.361903 29509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:28.361941 29509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:28.363898 29509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:28.364207 29509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:28.364456 29509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:28.370977 29514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.370958 29517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.371170 29515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:28.371469 29509 server_base.cc:1034] running on GCE node
I20260619 02:49:28.371702 29509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:28.371986 29509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:28.373234 29509 hybrid_clock.cc:648] HybridClock initialized: now 1781837368373209 us; error 41 us; skew 500 ppm
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:28.376802 29509 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:28.377491 29509 webserver.cc:460] Webserver started at http://127.26.98.1:43821/ using document root <none> and password file <none>
I20260619 02:49:28.377771 29509 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:28.377840 29509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:28.377998 29509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:28.379467 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "94e91948b8e64f4cb5199ab1e6faecda"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "01122a4196586df45f59f6ff83479d45"
server_key_iv: "3b3e7a99c36a93159a8f3a5d9c912f2a"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.379892 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "94e91948b8e64f4cb5199ab1e6faecda"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "01122a4196586df45f59f6ff83479d45"
server_key_iv: "3b3e7a99c36a93159a8f3a5d9c912f2a"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.381639 29509 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:28.382807 29524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.383060 29509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:28.383198 29509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "94e91948b8e64f4cb5199ab1e6faecda"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "01122a4196586df45f59f6ff83479d45"
server_key_iv: "3b3e7a99c36a93159a8f3a5d9c912f2a"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.383291 29509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:28.399086 29509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:28.402177 29509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:28.402426 29509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:28.402712 29509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:28.403157 29509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:28.403206 29509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.403244 29509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:28.403267 29509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.410420 29509 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:39393
I20260619 02:49:28.410485 29637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:39393 every 8 connection(s)
I20260619 02:49:28.410867 29509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:28.415899 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29509
I20260619 02:49:28.416119 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:28.416287 27016 external_mini_cluster.cc:1428] Setting key 2b38006bbc7247de7573dcd5a96db76f
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:28.425266 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.412048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:58543 (local address 127.26.98.62:34041)
0619 02:49:28.412107 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.412109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.412394 (+ 285us) server_negotiation.cc:407] Connection header received
0619 02:49:28.413034 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.413036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.413050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.413107 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.414830 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.415495 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.417052 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.417322 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.417405 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.421219 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.421221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.421222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:28.421233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.423255 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.423913 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.423914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.423916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.424001 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.424325 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.424326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.424326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.424520 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.424596 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.425008 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.425142 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:28.425411 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.411993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34041 (local address 127.26.98.1:58543)
0619 02:49:28.412221 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:28.412273 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:28.412915 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:28.413194 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:28.413199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:28.414299 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:28.414588 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.414598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.415630 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.415631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.416875 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.416883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.417460 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.417461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.417729 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.418858 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:28.418907 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:28.421060 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:28.423446 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.423461 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.423464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.423812 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.424105 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.424106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.424106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.424257 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.424645 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:28.424654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:28.424848 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 02:49:28.424970 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:49:28.425171 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260619 02:49:28.426287 29638 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34041
I20260619 02:49:28.426433 29638 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:28.426687 29638 heartbeater.cc:511] Master 127.26.98.62:34041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:49:28.427395 29447 ts_manager.cc:194] Registered new tserver with Master: 94e91948b8e64f4cb5199ab1e6faecda (127.26.98.1:39393)
I20260619 02:49:28.428115 29447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:58543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:28.451510 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:28.550861 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:28.551198 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:28.551230 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:28.552708 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:28.552765 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:28.552799 29643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:28.554544 29643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:28.554834 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:28.555147 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:28.560978 29651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.560978 29649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.560999 29648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:28.561038 29643 server_base.cc:1034] running on GCE node
I20260619 02:49:28.561465 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:28.561789 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:28.562983 29643 hybrid_clock.cc:648] HybridClock initialized: now 1781837368562952 us; error 49 us; skew 500 ppm
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:28.566226 29643 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:28.566820 29643 webserver.cc:460] Webserver started at http://127.26.98.2:45023/ using document root <none> and password file <none>
I20260619 02:49:28.567106 29643 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:28.567167 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:28.567322 29643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:28.568593 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "dabe55f460774d6cb93ef256682b5181"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "3503320c3b26895034cc890a5d9cc5ef"
server_key_iv: "16c9d0089c43e0032c894482b62d4859"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.569001 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "dabe55f460774d6cb93ef256682b5181"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "3503320c3b26895034cc890a5d9cc5ef"
server_key_iv: "16c9d0089c43e0032c894482b62d4859"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.570564 29643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:28.571550 29658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.571791 29643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:28.571900 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "dabe55f460774d6cb93ef256682b5181"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "3503320c3b26895034cc890a5d9cc5ef"
server_key_iv: "16c9d0089c43e0032c894482b62d4859"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.571974 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:28.589524 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:28.592159 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:28.592311 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:28.592552 29643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:28.592952 29643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:28.592989 29643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.593019 29643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:28.593043 29643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.600175 29643 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:37067
I20260619 02:49:28.600291 29771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:37067 every 8 connection(s)
I20260619 02:49:28.600646 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:28.608682 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29643
I20260619 02:49:28.608822 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:28.609046 27016 external_mini_cluster.cc:1428] Setting key 1f291826110ca37a1ee6a32077b6efc5
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:28.613875 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.601855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:48475 (local address 127.26.98.62:34041)
0619 02:49:28.601918 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.601920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.602178 (+ 258us) server_negotiation.cc:407] Connection header received
0619 02:49:28.602793 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.602794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.602808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.602918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.604564 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.605135 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.606473 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.606692 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.606770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.609958 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.609960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.609960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:28.609970 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.611953 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.612716 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.612718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.612719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.612807 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.613100 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.613102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.613102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.613244 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.613312 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.613652 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.613757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:28.614087 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.601823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34041 (local address 127.26.98.2:48475)
0619 02:49:28.602013 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:28.602067 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:28.602667 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:28.603008 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:28.603013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:28.604042 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:28.604370 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.604378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.605283 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.605284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.606279 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.606292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.606813 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.606814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.607035 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.607732 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:28.607742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:28.609854 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:28.612086 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.612100 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.612104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.612603 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.612893 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.612893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.612894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.613037 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.613309 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:28.613317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:28.613507 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:49:28.613617 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:49:28.613827 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260619 02:49:28.614655 29772 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34041
I20260619 02:49:28.614796 29772 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:28.615100 29772 heartbeater.cc:511] Master 127.26.98.62:34041 requested a full tablet report, sending...
I20260619 02:49:28.615666 29447 ts_manager.cc:194] Registered new tserver with Master: dabe55f460774d6cb93ef256682b5181 (127.26.98.2:37067)
I20260619 02:49:28.616230 29447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:48475
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:28.643709 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:28.743837 29777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:28.744122 29777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:28.744151 29777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:28.745596 29777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:28.745666 29777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:28.745703 29777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:28.747534 29777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:28.747871 29777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:28.748142 29777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:28.754647 29782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.754642 29783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.754642 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:28.755127 29777 server_base.cc:1034] running on GCE node
I20260619 02:49:28.755420 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:28.755740 29777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:28.756917 29777 hybrid_clock.cc:648] HybridClock initialized: now 1781837368756895 us; error 38 us; skew 500 ppm
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:28.760854 29777 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:28.761587 29777 webserver.cc:460] Webserver started at http://127.26.98.3:33413/ using document root <none> and password file <none>
I20260619 02:49:28.761883 29777 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:28.761957 29777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:28.762301 29777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:28.763928 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "4ff5903cfcec42c08437810699ee0760"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "9cc96fd0d1de88f2f4f86d31c936095c"
server_key_iv: "0f3c3449807fc36edf8ed2e811c5551b"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.764416 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "4ff5903cfcec42c08437810699ee0760"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "9cc96fd0d1de88f2f4f86d31c936095c"
server_key_iv: "0f3c3449807fc36edf8ed2e811c5551b"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.766252 29777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 02:49:28.767575 29792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.767872 29777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:28.768005 29777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "4ff5903cfcec42c08437810699ee0760"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-0f9d"
server_key: "9cc96fd0d1de88f2f4f86d31c936095c"
server_key_iv: "0f3c3449807fc36edf8ed2e811c5551b"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.768131 29777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:28.793357 29777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:28.796507 29777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:28.796695 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:28.796996 29777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:28.797394 29777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:28.797439 29777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.797475 29777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:28.797492 29777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.804762 29777 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:45393
I20260619 02:49:28.804827 29905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:45393 every 8 connection(s)
I20260619 02:49:28.805251 29777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:28.811225 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29777
I20260619 02:49:28.811342 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:28.811545 27016 external_mini_cluster.cc:1428] Setting key b6e345fafbf4a2d8ded2471be31c2376
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:28.817509 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.806267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59967 (local address 127.26.98.62:34041)
0619 02:49:28.806373 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.806374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.806594 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:49:28.807221 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.807223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.807236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.807319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.808673 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.809268 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.810579 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.810874 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.811080 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.813834 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.813836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.813836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:28.813846 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.815706 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.816269 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.816270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.816271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.816343 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:28.816729 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:28.816730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:28.816731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:28.816888 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.816953 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.817329 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.817424 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:49:28.817723 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.806179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34041 (local address 127.26.98.3:59967)
0619 02:49:28.806398 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:28.806454 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:28.807079 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:28.807447 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:28.807453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:28.808221 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:28.808506 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.808514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.809405 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.809406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.810395 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:28.810398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:28.811087 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:28.811088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:28.811264 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:28.811944 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:28.811957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:28.813661 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:28.815825 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.815843 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.815847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.816167 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.816482 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:28.816483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:28.816483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:28.816637 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:28.816960 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:28.816967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:28.817157 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:49:28.817277 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:49:28.817463 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260619 02:49:28.818292 29906 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34041
I20260619 02:49:28.818406 29906 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:28.818609 29906 heartbeater.cc:511] Master 127.26.98.62:34041 requested a full tablet report, sending...
I20260619 02:49:28.819090 29447 ts_manager.cc:194] Registered new tserver with Master: 4ff5903cfcec42c08437810699ee0760 (127.26.98.3:45393)
I20260619 02:49:28.819490 29447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:59967
I20260619 02:49:28.824350 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:49:28.839735 29486 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:28.838033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.26.98.62:34041)
0619 02:49:28.838099 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.838101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.838200 (+ 99us) server_negotiation.cc:407] Connection header received
0619 02:49:28.838279 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.838280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.838292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.838358 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.839327 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56222: BlockingRecv error: recv got EOF from 127.0.0.1:56222 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:28.840471 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29509
I20260619 02:49:28.845268 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29643
I20260619 02:49:28.850067 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29777
I20260619 02:49:28.855069 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29415
2026-06-19T02:49:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2001 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29918](info): set up 2 sockets
Jun 19 02:49:28 dist-test-slave-0f9d krb5kdc[29918](info): commencing operation
krb5kdc: starting...
W20260619 02:49:29.851114 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:29 dist-test-slave-0f9d krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:29.991727 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:41189
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:30.090233 29931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:30.090535 29931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:30.090565 29931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:30.092257 29931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:30.092342 29931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:30.092357 29931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:30.092370 29931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:30.092381 29931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:30.094787 29931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:41189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:30.095189 29931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:30.095474 29931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:49:30.101161 29931 server_base.cc:1034] running on GCE node
W20260619 02:49:30.101125 29939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:30.101128 29936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:30.101128 29937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:30.101646 29931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:30.102026 29931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:30.103235 29931 hybrid_clock.cc:648] HybridClock initialized: now 1781837370103201 us; error 53 us; skew 500 ppm
Jun 19 02:49:30 dist-test-slave-0f9d krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837370, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:30.112733 29931 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:30.113595 29931 webserver.cc:460] Webserver started at http://127.26.98.62:43947/ using document root <none> and password file <none>
I20260619 02:49:30.113896 29931 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:30.113958 29931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:30.114104 29931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:30.115561 29931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "fcf5e8a54d6c450ab331e1e633aac4bd"
format_stamp: "Formatted at 2026-06-19 02:49:30 on dist-test-slave-0f9d"
server_key: "50aa95f8a135fed35f588ad7427498dc"
server_key_iv: "d65e2d4a43c73018f67600be838416ef"
server_key_version: "encryptionkey@0"
I20260619 02:49:30.115993 29931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "fcf5e8a54d6c450ab331e1e633aac4bd"
format_stamp: "Formatted at 2026-06-19 02:49:30 on dist-test-slave-0f9d"
server_key: "50aa95f8a135fed35f588ad7427498dc"
server_key_iv: "d65e2d4a43c73018f67600be838416ef"
server_key_version: "encryptionkey@0"
I20260619 02:49:30.117691 29931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:30.118829 29946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:30.119064 29931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:30.119179 29931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "fcf5e8a54d6c450ab331e1e633aac4bd"
format_stamp: "Formatted at 2026-06-19 02:49:30 on dist-test-slave-0f9d"
server_key: "50aa95f8a135fed35f588ad7427498dc"
server_key_iv: "d65e2d4a43c73018f67600be838416ef"
server_key_version: "encryptionkey@0"
I20260619 02:49:30.119269 29931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:30.141734 29931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:30.145289 29931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:30.145607 29931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:30.151297 29931 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:41189
I20260619 02:49:30.151561 29998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:41189 every 8 connection(s)
I20260619 02:49:30.151808 29931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:30.152874 29999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:30.156169 29999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd: Bootstrap starting.
I20260619 02:49:30.156906 29999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:30.157343 29999 log.cc:824] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:30.158355 29999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd: No bootstrap required, opened a new log
I20260619 02:49:30.159624 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 29931
I20260619 02:49:30.159747 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:30.160046 27016 external_mini_cluster.cc:1428] Setting key 7a80bfd28b1fd4f97572a0fd685eb2f6
I20260619 02:49:30.160533 29999 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } }
I20260619 02:49:30.160769 29999 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:30.160840 29999 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf5e8a54d6c450ab331e1e633aac4bd, State: Initialized, Role: FOLLOWER
I20260619 02:49:30.161017 29999 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } }
I20260619 02:49:30.161118 29999 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:30.161162 29999 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:30.161226 29999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:30.162289 29999 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } }
I20260619 02:49:30.162446 29999 leader_election.cc:302] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcf5e8a54d6c450ab331e1e633aac4bd; no voters:
I20260619 02:49:30.162766 29999 leader_election.cc:288] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:30.162847 30004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:30.163054 30004 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [term 1 LEADER]: Becoming Leader. State: Replica: fcf5e8a54d6c450ab331e1e633aac4bd, State: Running, Role: LEADER
I20260619 02:49:30.163182 29999 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:30.163223 30004 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } }
I20260619 02:49:30.163714 30005 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } } }
I20260619 02:49:30.163838 30005 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:30.163724 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcf5e8a54d6c450ab331e1e633aac4bd. Latest consensus state: current_term: 1 leader_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf5e8a54d6c450ab331e1e633aac4bd" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 41189 } } }
I20260619 02:49:30.164187 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:30.164532 30012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:30.165333 30012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:30.167253 30012 catalog_manager.cc:1347] Generated new cluster ID: 0209e4affcec49249656f37517c7d36d
I20260619 02:49:30.167316 30012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:30 dist-test-slave-0f9d krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:30.174769 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:30.161110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.26.98.62:41189)
0619 02:49:30.161351 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:30.161368 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:30.161416 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:30.161594 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:30.161605 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:30.161690 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:30.161891 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:30.162732 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.164357 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.165621 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.166308 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.166449 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:30.169778 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:30.169787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:30.169788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:30.169809 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:30.172510 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.173130 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.173137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.173137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.173330 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.173549 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.173550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.173550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.173990 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:30.174209 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:30.174301 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 02:49:30.174490 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":120,"threads_started":1}
I20260619 02:49:30.176008 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:30.198534 30012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:30.199370 30012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:30.201299 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:30.196018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.26.98.62:41189)
0619 02:49:30.196112 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:30.196114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:30.196639 (+ 525us) server_negotiation.cc:407] Connection header received
0619 02:49:30.196720 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:30.196721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:30.196739 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:30.196808 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:30.197346 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.198152 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.199393 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.199722 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.199793 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:30.200678 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:30.200680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:30.200680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:30.200970 (+ 290us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:30.200997 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:30.201006 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:30.201073 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:30.201130 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 02:49:30.201202 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:30.202353 30012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcf5e8a54d6c450ab331e1e633aac4bd: Generated new TSK 0
I20260619 02:49:30.202566 30012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:30.210402 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 29931
2026-06-19T02:49:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1353 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:30 dist-test-slave-0f9d krb5kdc[30032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:30 dist-test-slave-0f9d krb5kdc[30032](info): set up 2 sockets
Jun 19 02:49:30 dist-test-slave-0f9d krb5kdc[30032](info): commencing operation
krb5kdc: starting...
W20260619 02:49:31.223114 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:31.361943 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39655
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:31.461607 30045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:31.462081 30045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:31.462126 30045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:31.464722 30045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:31.464831 30045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:31.464854 30045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:31.464874 30045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:31.464890 30045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:31.468633 30045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:31.469018 30045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:31.469264 30045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:31.475334 30051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.475334 30050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.475914 30053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:31.475998 30045 server_base.cc:1034] running on GCE node
I20260619 02:49:31.476346 30045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:31.476857 30045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:31.478062 30045 hybrid_clock.cc:648] HybridClock initialized: now 1781837371478032 us; error 59 us; skew 500 ppm
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:31.481222 30045 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:31.481813 30045 webserver.cc:460] Webserver started at http://127.26.98.62:45899/ using document root <none> and password file <none>
I20260619 02:49:31.482062 30045 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:31.482118 30045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:31.482285 30045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:31.483717 30045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "b593ebf5ff5b48b2a84f64f9d59297d1"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "b9cd2bec6103b50a9a0f0c376d7a8118"
server_key_iv: "6b507cb500f2e70305229e63dd030208"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.484115 30045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "b593ebf5ff5b48b2a84f64f9d59297d1"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "b9cd2bec6103b50a9a0f0c376d7a8118"
server_key_iv: "6b507cb500f2e70305229e63dd030208"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.485715 30045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:49:31.486811 30060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.487082 30045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:31.487226 30045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "b593ebf5ff5b48b2a84f64f9d59297d1"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "b9cd2bec6103b50a9a0f0c376d7a8118"
server_key_iv: "6b507cb500f2e70305229e63dd030208"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.487321 30045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:31.519531 30045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:31.522063 30045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:31.522277 30045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:31.527614 30045 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:39655
I20260619 02:49:31.527699 30112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:39655 every 8 connection(s)
I20260619 02:49:31.528115 30045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:31.529331 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30045
I20260619 02:49:31.529410 30113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:31.529448 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:31.529659 27016 external_mini_cluster.cc:1428] Setting key 93e701c64b299f20b025261d4750ab32
I20260619 02:49:31.533499 30113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1: Bootstrap starting.
I20260619 02:49:31.534556 30113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:31.535001 30113 log.cc:824] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:31.536021 30113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1: No bootstrap required, opened a new log
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:31.538980 30113 raft_consensus.cc:348] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } }
I20260619 02:49:31.539220 30113 raft_consensus.cc:374] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:31.539264 30113 raft_consensus.cc:729] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b593ebf5ff5b48b2a84f64f9d59297d1, State: Initialized, Role: FOLLOWER
I20260619 02:49:31.539434 30113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } }
I20260619 02:49:31.539528 30113 raft_consensus.cc:388] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:31.539574 30113 raft_consensus.cc:482] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:31.539637 30113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:31.540393 30113 raft_consensus.cc:504] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } }
I20260619 02:49:31.540527 30113 leader_election.cc:302] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b593ebf5ff5b48b2a84f64f9d59297d1; no voters:
I20260619 02:49:31.540797 30113 leader_election.cc:288] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:31.540850 30118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:31.541047 30118 raft_consensus.cc:686] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [term 1 LEADER]: Becoming Leader. State: Replica: b593ebf5ff5b48b2a84f64f9d59297d1, State: Running, Role: LEADER
I20260619 02:49:31.541203 30113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:31.541211 30118 consensus_queue.cc:237] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } }
I20260619 02:49:31.541700 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b593ebf5ff5b48b2a84f64f9d59297d1. Latest consensus state: current_term: 1 leader_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } } }
I20260619 02:49:31.541847 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:31.542109 30126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:31.542773 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b593ebf5ff5b48b2a84f64f9d59297d1" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39655 } } }
I20260619 02:49:31.542867 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:31.543123 30126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:31.544162 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:31.530317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.26.98.62:39655)
0619 02:49:31.530533 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:31.530543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:31.530591 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:31.531281 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:31.531292 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:31.531369 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:31.531524 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:31.532411 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.533607 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.535428 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.536066 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.536224 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:31.539526 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:31.539533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:31.539533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:31.539550 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:31.541729 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.542324 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.542330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.542330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.542468 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.542722 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.542722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.542723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.543141 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:31.543787 (+ 646us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:31.543856 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:49:31.543987 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260619 02:49:31.545253 30126 catalog_manager.cc:1347] Generated new cluster ID: d533084332534b06af6f699b77330f4a
I20260619 02:49:31.545317 30126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:31.563788 30126 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:31.564857 30126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:31.570155 30126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b593ebf5ff5b48b2a84f64f9d59297d1: Generated new TSK 0
I20260619 02:49:31.570472 30126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:31.611009 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:31.726204 30139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:31.726521 30139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:31.726553 30139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:31.728175 30139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:31.728260 30139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:31.728300 30139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:31.730434 30139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:31.730782 30139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:31.731103 30139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:31.737739 30147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.737811 30145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.737761 30144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:31.738425 30139 server_base.cc:1034] running on GCE node
I20260619 02:49:31.738737 30139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:31.739094 30139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:31.740334 30139 hybrid_clock.cc:648] HybridClock initialized: now 1781837371740300 us; error 32 us; skew 500 ppm
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:31.744381 30139 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:31.745049 30139 webserver.cc:460] Webserver started at http://127.26.98.1:35483/ using document root <none> and password file <none>
I20260619 02:49:31.745355 30139 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:31.745419 30139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:31.745580 30139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:31.747040 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "aad2bba36be54981aa6ca498c9c884be"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "e2967aa7f2269635345b377dc0663636"
server_key_iv: "aa8caf16b2108a4c3a81ad9c692af91b"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.747520 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "aad2bba36be54981aa6ca498c9c884be"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "e2967aa7f2269635345b377dc0663636"
server_key_iv: "aa8caf16b2108a4c3a81ad9c692af91b"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.749441 30139 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:31.750561 30154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.750849 30139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:31.751000 30139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "aad2bba36be54981aa6ca498c9c884be"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "e2967aa7f2269635345b377dc0663636"
server_key_iv: "aa8caf16b2108a4c3a81ad9c692af91b"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.751085 30139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:31.770107 30139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:31.773500 30139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:31.773689 30139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:31.774019 30139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:31.774459 30139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:31.774498 30139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.774526 30139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:31.774537 30139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.781800 30139 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:38397
I20260619 02:49:31.781934 30267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:38397 every 8 connection(s)
I20260619 02:49:31.782300 30139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:31.790023 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30139
I20260619 02:49:31.790158 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:31.790406 27016 external_mini_cluster.cc:1428] Setting key c8bc508dd80cbc1f1e711d57ea4c1c1c
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:31.795354 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:31.783468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:37623 (local address 127.26.98.62:39655)
0619 02:49:31.783568 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:31.783570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:31.783865 (+ 295us) server_negotiation.cc:407] Connection header received
0619 02:49:31.784696 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:31.784698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:31.784716 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:31.784751 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:31.786411 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.787165 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.788329 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.788575 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.788633 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:31.791428 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:31.791429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:31.791430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:31.791439 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:31.793062 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.793942 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.793945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.793945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.794063 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.794331 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.794332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.794332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.794564 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:31.794673 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:31.795106 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:49:31.795240 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:31.795562 30270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:31.783470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39655 (local address 127.26.98.1:37623)
0619 02:49:31.783661 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:31.783714 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:31.784527 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:31.784987 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:31.784992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:31.785899 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:31.786282 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:31.786292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.787271 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.787272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:31.788210 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:31.788213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.788657 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.788658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:31.788819 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:31.789478 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:31.789488 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:31.791320 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:31.793182 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:31.793192 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:31.793195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:31.793762 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:31.794136 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:31.794137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:31.794137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:31.794268 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:31.794694 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:31.794705 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:31.794960 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 02:49:31.795078 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 02:49:31.795296 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260619 02:49:31.796156 30268 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39655
I20260619 02:49:31.796314 30268 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:31.796582 30268 heartbeater.cc:511] Master 127.26.98.62:39655 requested a full tablet report, sending...
I20260619 02:49:31.797535 30077 ts_manager.cc:194] Registered new tserver with Master: aad2bba36be54981aa6ca498c9c884be (127.26.98.1:38397)
I20260619 02:49:31.798586 30077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:37623
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:31.818581 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:31.917580 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:31.917909 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:31.917940 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:31.919472 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:31.919556 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:31.919595 30273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:31.921482 30273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:31.921808 30273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:31.922082 30273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:31.928455 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.928455 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.928481 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:31.928874 30273 server_base.cc:1034] running on GCE node
I20260619 02:49:31.929137 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:31.929436 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:31.930671 30273 hybrid_clock.cc:648] HybridClock initialized: now 1781837371930632 us; error 40 us; skew 500 ppm
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:31.934317 30273 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:31.934968 30273 webserver.cc:460] Webserver started at http://127.26.98.2:41313/ using document root <none> and password file <none>
I20260619 02:49:31.935232 30273 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:31.935285 30273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:31.935428 30273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:31.936668 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "5ab8f153a00a4e56a620ed4ee2453205"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "db0421bd1047d4f250024a583defeef1"
server_key_iv: "29fa10190a03e27e44d9fd8139b2f5d6"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.937049 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "5ab8f153a00a4e56a620ed4ee2453205"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "db0421bd1047d4f250024a583defeef1"
server_key_iv: "29fa10190a03e27e44d9fd8139b2f5d6"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.938495 30273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:31.939421 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.939637 30273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:31.939731 30273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "5ab8f153a00a4e56a620ed4ee2453205"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-0f9d"
server_key: "db0421bd1047d4f250024a583defeef1"
server_key_iv: "29fa10190a03e27e44d9fd8139b2f5d6"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.939801 30273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:31.952726 30273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:31.956339 30273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:31.956528 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:31.956809 30273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:31.957198 30273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:31.957235 30273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.957270 30273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:31.957294 30273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:31.964550 30273 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40131
I20260619 02:49:31.964634 30401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40131 every 8 connection(s)
I20260619 02:49:31.965005 30273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:31.965487 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30273
I20260619 02:49:31.965608 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:31.965809 27016 external_mini_cluster.cc:1428] Setting key f12e0b973a6dfed87a28607217c5c4db
Jun 19 02:49:31 dist-test-slave-0f9d krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:49:31.978188 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:31.965971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:33971 (local address 127.26.98.62:39655)
0619 02:49:31.966035 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:31.966037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:31.966311 (+ 274us) server_negotiation.cc:407] Connection header received
0619 02:49:31.966974 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:31.966975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:31.966990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:31.967034 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:31.968696 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.969518 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.971271 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.971508 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.971575 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:31.974774 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:31.974776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:31.974777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:31.974786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:31.976527 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.977083 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.977084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.977084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.977147 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:31.977429 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:31.977430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:31.977431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:31.977567 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:31.977629 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:31.977967 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 02:49:31.978076 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
I20260619 02:49:31.978423 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:31.965935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39655 (local address 127.26.98.2:33971)
0619 02:49:31.966163 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:31.966219 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:31.966846 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:31.967133 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:31.967138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:31.968125 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:31.968533 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:31.968544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.969660 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.969661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:31.971125 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:31.971130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:31.971593 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:31.971594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:31.971769 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:31.972481 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:31.972492 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:31.974614 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:31.976659 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:31.976676 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:31.976680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:31.976995 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:31.977222 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:31.977223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:31.977223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:31.977363 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:31.977633 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:31.977641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:31.977818 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:49:31.977923 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:49:31.978134 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260619 02:49:31.978941 30402 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39655
I20260619 02:49:31.979055 30402 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:31.979278 30402 heartbeater.cc:511] Master 127.26.98.62:39655 requested a full tablet report, sending...
I20260619 02:49:31.979732 30077 ts_manager.cc:194] Registered new tserver with Master: 5ab8f153a00a4e56a620ed4ee2453205 (127.26.98.2:40131)
I20260619 02:49:31.980156 30077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:33971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:31.994728 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:32.093322 30407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:32.093583 30407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:32.093616 30407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:32.095091 30407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:32.095152 30407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:32.095189 30407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:32.097613 30407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:32.097950 30407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:32.098222 30407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:32.104728 30412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:32.104805 30407 server_base.cc:1034] running on GCE node
W20260619 02:49:32.104728 30413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:32.104732 30415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:32.105167 30407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:32.105461 30407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:32.106650 30407 hybrid_clock.cc:648] HybridClock initialized: now 1781837372106623 us; error 42 us; skew 500 ppm
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837372, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:32.110394 30407 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:32.111546 30407 webserver.cc:460] Webserver started at http://127.26.98.3:42341/ using document root <none> and password file <none>
I20260619 02:49:32.111841 30407 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:32.111894 30407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:32.112020 30407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:32.113318 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "046e9cf218c04ce0b8e9ea0fd9544c4b"
format_stamp: "Formatted at 2026-06-19 02:49:32 on dist-test-slave-0f9d"
server_key: "d82c87e877fda718b0cd441a502dbb1b"
server_key_iv: "883d828b6a7b34b0feb19cd70c421a2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:32.113695 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "046e9cf218c04ce0b8e9ea0fd9544c4b"
format_stamp: "Formatted at 2026-06-19 02:49:32 on dist-test-slave-0f9d"
server_key: "d82c87e877fda718b0cd441a502dbb1b"
server_key_iv: "883d828b6a7b34b0feb19cd70c421a2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:32.115927 30407 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:49:32.118219 30422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:32.118575 30407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:49:32.118722 30407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "046e9cf218c04ce0b8e9ea0fd9544c4b"
format_stamp: "Formatted at 2026-06-19 02:49:32 on dist-test-slave-0f9d"
server_key: "d82c87e877fda718b0cd441a502dbb1b"
server_key_iv: "883d828b6a7b34b0feb19cd70c421a2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:32.118840 30407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:32.133158 30407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:32.136437 30407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:32.136591 30407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:32.136835 30407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:32.137148 30407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:32.137180 30407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:32.137207 30407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:32.137220 30407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:32.145016 30407 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:40663
I20260619 02:49:32.145113 30535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:40663 every 8 connection(s)
I20260619 02:49:32.145483 30407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:32.152137 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30407
I20260619 02:49:32.152253 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:32.152441 27016 external_mini_cluster.cc:1428] Setting key f206adc25dd78d329ae76e307a079131
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837372, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:32.160646 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:32.146356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:44617 (local address 127.26.98.62:39655)
0619 02:49:32.146446 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:32.146448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:32.146640 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:49:32.147343 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:32.147344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:32.147359 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:32.147414 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:32.149376 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.150084 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.151780 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.152015 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.152097 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:32.155618 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:32.155621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:32.155621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:32.155630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:32.158027 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:32.158909 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:32.158910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:32.158911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:32.159001 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:32.159463 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:32.159465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:32.159465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:32.159636 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:32.159715 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:32.160406 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 02:49:32.160536 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:49:32.160755 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:32.146312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39655 (local address 127.26.98.3:44617)
0619 02:49:32.146493 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:32.146546 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:32.147224 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:32.147508 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:32.147512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:32.148250 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:32.149079 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:32.149090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.150269 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.150271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:32.151543 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:32.151550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.152134 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.152135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:32.152351 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:32.153281 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:32.153301 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:32.155492 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:32.158230 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:32.158245 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:32.158248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:32.158744 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:32.159080 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:32.159081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:32.159081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:32.159309 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:32.159736 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:32.159744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:32.159955 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 02:49:32.160224 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 02:49:32.160505 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260619 02:49:32.161403 30536 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39655
I20260619 02:49:32.161600 30536 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:32.161923 30536 heartbeater.cc:511] Master 127.26.98.62:39655 requested a full tablet report, sending...
I20260619 02:49:32.162618 30077 ts_manager.cc:194] Registered new tserver with Master: 046e9cf218c04ce0b8e9ea0fd9544c4b (127.26.98.3:40663)
I20260619 02:49:32.163126 30077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:44617
I20260619 02:49:32.165462 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837372, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837372, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:32.197407 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:32.186554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.26.98.62:39655)
0619 02:49:32.186658 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:32.186661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:32.186678 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:32.186777 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:32.186778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:32.186790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:32.186872 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:32.187906 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.188491 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.189327 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.189534 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.189603 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:32.193040 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:32.193043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:32.193043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:32.193057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:32.195160 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:32.195731 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:32.195734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:32.195734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:32.195832 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:32.196172 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:32.196173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:32.196173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:32.196462 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:32.196580 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:32.197207 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 02:49:32.197316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 02:49:32.197793 30077 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33294
I20260619 02:49:32.198719 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30139
I20260619 02:49:32.203986 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30273
I20260619 02:49:32.209316 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30407
I20260619 02:49:32.214483 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30045
2026-06-19T02:49:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2005 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30548](info): set up 2 sockets
Jun 19 02:49:32 dist-test-slave-0f9d krb5kdc[30548](info): commencing operation
krb5kdc: starting...
W20260619 02:49:33.203491 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:33.347108 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46457
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:33.448199 30561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:33.448539 30561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:33.448580 30561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:33.450099 30561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:33.450171 30561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:33.450192 30561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:33.450210 30561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:33.450225 30561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:33.452749 30561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:33.453356 30561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:33.453706 30561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:33.461046 30569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.461035 30567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.461035 30566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:33.462075 30561 server_base.cc:1034] running on GCE node
I20260619 02:49:33.462435 30561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:33.462962 30561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:33.464181 30561 hybrid_clock.cc:648] HybridClock initialized: now 1781837373464142 us; error 62 us; skew 500 ppm
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:33.468596 30561 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:33.469367 30561 webserver.cc:460] Webserver started at http://127.26.98.62:37645/ using document root <none> and password file <none>
I20260619 02:49:33.469735 30561 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:33.469820 30561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:33.469993 30561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:33.471946 30561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "250952c0ff82498eaa4917772b19789f"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "702958c1ad5c83749c2dc4bac132dff7"
server_key_iv: "3e4aa0b311ec65f3eb199dbbf3f76fa8"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.472536 30561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "250952c0ff82498eaa4917772b19789f"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "702958c1ad5c83749c2dc4bac132dff7"
server_key_iv: "3e4aa0b311ec65f3eb199dbbf3f76fa8"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.474994 30561 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 02:49:33.476123 30576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.476382 30561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:33.476516 30561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "250952c0ff82498eaa4917772b19789f"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "702958c1ad5c83749c2dc4bac132dff7"
server_key_iv: "3e4aa0b311ec65f3eb199dbbf3f76fa8"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.476606 30561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:33.493162 30561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:33.496389 30561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:33.496555 30561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:33.501319 30561 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:46457
I20260619 02:49:33.501385 30628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:46457 every 8 connection(s)
I20260619 02:49:33.501945 30561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:33.502861 30629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:33.504518 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30561
I20260619 02:49:33.504648 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:33.504839 27016 external_mini_cluster.cc:1428] Setting key 5a0372eb8776a95eb607ee90eb18f5dd
I20260619 02:49:33.506402 30629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: Bootstrap starting.
I20260619 02:49:33.507470 30629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:33.507995 30629 log.cc:824] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:33.508838 30629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: No bootstrap required, opened a new log
I20260619 02:49:33.511579 30629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } }
I20260619 02:49:33.511804 30629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:33.511843 30629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250952c0ff82498eaa4917772b19789f, State: Initialized, Role: FOLLOWER
I20260619 02:49:33.512009 30629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } }
I20260619 02:49:33.512102 30629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:33.512146 30629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:33.512213 30629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:33.513285 30629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } }
I20260619 02:49:33.513448 30629 leader_election.cc:302] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 250952c0ff82498eaa4917772b19789f; no voters:
I20260619 02:49:33.513688 30629 leader_election.cc:288] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:33.513768 30634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:33.514111 30629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:33.514122 30634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [term 1 LEADER]: Becoming Leader. State: Replica: 250952c0ff82498eaa4917772b19789f, State: Running, Role: LEADER
I20260619 02:49:33.514490 30634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } }
I20260619 02:49:33.515836 30635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "250952c0ff82498eaa4917772b19789f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } } }
I20260619 02:49:33.515985 30635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:33.516822 30648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:33.516907 30648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:33.517627 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 250952c0ff82498eaa4917772b19789f. Latest consensus state: current_term: 1 leader_uuid: "250952c0ff82498eaa4917772b19789f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250952c0ff82498eaa4917772b19789f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46457 } } }
I20260619 02:49:33.517751 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:33.517932 30649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:33.518214 30649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:33.519946 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.505943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.26.98.62:46457)
0619 02:49:33.506203 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:33.506215 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:33.506268 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:49:33.507018 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:33.507031 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:33.507106 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:33.507431 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:33.508210 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.509290 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.510133 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.510673 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.510807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:33.514132 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:33.514141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:33.514141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:33.514163 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:33.517284 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.517972 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.517977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.517977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.518125 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.518420 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.518421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.518421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.519147 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:33.519359 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:33.519507 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:49:33.519699 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":122,"threads_started":1}
I20260619 02:49:33.521354 30649 catalog_manager.cc:1347] Generated new cluster ID: 7b4e64506e5e4cb8a6ad2ef49a73e960
I20260619 02:49:33.521440 30649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:33.533120 30649 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:33.533833 30649 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:33.540660 30649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 250952c0ff82498eaa4917772b19789f: Generated new TSK 0
I20260619 02:49:33.541007 30649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:33.576124 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:33.683372 30655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:33.683676 30655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:33.683708 30655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:33.685425 30655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:33.685529 30655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:33.685570 30655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:33.687724 30655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:33.688082 30655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:33.688372 30655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:33.694489 30663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.694523 30661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.694489 30660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:33.694952 30655 server_base.cc:1034] running on GCE node
I20260619 02:49:33.695212 30655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:33.695525 30655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:33.696700 30655 hybrid_clock.cc:648] HybridClock initialized: now 1781837373696684 us; error 38 us; skew 500 ppm
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:33.700006 30655 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:33.700663 30655 webserver.cc:460] Webserver started at http://127.26.98.1:40125/ using document root <none> and password file <none>
I20260619 02:49:33.700919 30655 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:33.700976 30655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:33.701130 30655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:33.702495 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "0fd16898c99b45a0bcfc1fe2493ae237"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "d535c5381cf42c4a4e68ff261951a5e2"
server_key_iv: "e285f61104703831b9ed00a509be7c49"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.702924 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "0fd16898c99b45a0bcfc1fe2493ae237"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "d535c5381cf42c4a4e68ff261951a5e2"
server_key_iv: "e285f61104703831b9ed00a509be7c49"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.704440 30655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:33.705574 30670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.705823 30655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:33.705922 30655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "0fd16898c99b45a0bcfc1fe2493ae237"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "d535c5381cf42c4a4e68ff261951a5e2"
server_key_iv: "e285f61104703831b9ed00a509be7c49"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.705998 30655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:33.722298 30655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:33.724906 30655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:33.725087 30655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:33.725508 30655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:33.726094 30655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:33.726158 30655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.726197 30655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:33.726214 30655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.733333 30655 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:40437
I20260619 02:49:33.733397 30783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:40437 every 8 connection(s)
I20260619 02:49:33.733789 30655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:33.743553 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30655
I20260619 02:49:33.743682 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:33.743898 27016 external_mini_cluster.cc:1428] Setting key ff1fef1236de06606442d50c337b8fc8
I20260619 02:49:33.744417 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.734818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:55105 (local address 127.26.98.62:46457)
0619 02:49:33.734901 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:33.734904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:33.735001 (+ 97us) server_negotiation.cc:407] Connection header received
0619 02:49:33.736093 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:33.736095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:33.736106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:33.736200 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:33.737102 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.737719 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.738967 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.739181 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.739245 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:33.741685 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:33.741687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:33.741687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:33.741695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:33.742907 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.743278 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.743278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.743279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.743346 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.743623 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.743623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.743624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.743775 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:33.743827 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:33.744212 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 02:49:33.744318 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:33.744603 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.734630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46457 (local address 127.26.98.1:55105)
0619 02:49:33.734820 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:33.734873 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:33.735977 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:33.736270 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:33.736275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:33.736641 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:33.736940 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:33.736945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.737838 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.737839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:33.738835 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:33.738838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.739259 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.739260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:33.739377 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:33.739990 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:33.740000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:33.741587 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:33.742969 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:33.742975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:33.742978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:33.743218 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:33.743404 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:33.743405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:33.743406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:33.743559 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:33.743823 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:33.743831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:33.744025 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 02:49:33.744147 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:49:33.744336 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260619 02:49:33.745260 30784 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46457
I20260619 02:49:33.745385 30784 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:33.745628 30784 heartbeater.cc:511] Master 127.26.98.62:46457 requested a full tablet report, sending...
I20260619 02:49:33.746385 30593 ts_manager.cc:194] Registered new tserver with Master: 0fd16898c99b45a0bcfc1fe2493ae237 (127.26.98.1:40437)
I20260619 02:49:33.747130 30593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:55105
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:33.771970 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:33.874948 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:33.875268 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:33.875304 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:33.876838 30789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:33.876904 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:33.876938 30789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:33.878968 30789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:33.879330 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:33.879592 30789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:33.886224 30794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.886224 30795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.886368 30797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:33.887054 30789 server_base.cc:1034] running on GCE node
I20260619 02:49:33.887428 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:33.887802 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:33.889025 30789 hybrid_clock.cc:648] HybridClock initialized: now 1781837373888951 us; error 98 us; skew 500 ppm
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:33.892670 30789 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:33.893388 30789 webserver.cc:460] Webserver started at http://127.26.98.2:36145/ using document root <none> and password file <none>
I20260619 02:49:33.893692 30789 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:33.893764 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:33.893927 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:33.895388 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "e9ba6ad20c2f49ed8f99d839bc6e4a8d"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "188085943f42b04adc5c1994c413cd91"
server_key_iv: "aea63ac5e213132a99a56542d13df075"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.895817 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "e9ba6ad20c2f49ed8f99d839bc6e4a8d"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "188085943f42b04adc5c1994c413cd91"
server_key_iv: "aea63ac5e213132a99a56542d13df075"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.897480 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:33.898603 30804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.899041 30789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:33.899191 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "e9ba6ad20c2f49ed8f99d839bc6e4a8d"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-0f9d"
server_key: "188085943f42b04adc5c1994c413cd91"
server_key_iv: "aea63ac5e213132a99a56542d13df075"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.899289 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:33.917672 30789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:33.920826 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:33.921007 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:33.921300 30789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:33.921686 30789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:33.921725 30789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.921759 30789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:33.921784 30789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.930362 30789 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:36693
I20260619 02:49:33.930449 30917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:36693 every 8 connection(s)
I20260619 02:49:33.930922 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:33.939257 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30789
I20260619 02:49:33.939415 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:33.939616 27016 external_mini_cluster.cc:1428] Setting key 32aaafbe15689a60f67633beee39e7bb
Jun 19 02:49:33 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:33.945444 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.931849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:55997 (local address 127.26.98.62:46457)
0619 02:49:33.931917 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:33.931919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:33.932176 (+ 257us) server_negotiation.cc:407] Connection header received
0619 02:49:33.933316 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:33.933318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:33.933366 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:33.933457 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:33.934404 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.935297 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.936922 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.937143 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.937258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:33.940703 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:33.940706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:33.940706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:33.940719 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:33.942968 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.943694 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.943696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.943697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.943789 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:33.944347 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:33.944348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:33.944349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:33.944557 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:33.944669 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:33.945205 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 02:49:33.945346 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:49:33.945695 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.931816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46457 (local address 127.26.98.2:55997)
0619 02:49:33.931993 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:33.932045 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:33.933138 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:33.933586 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:33.933591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:33.933957 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:33.934261 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:33.934267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.935489 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.935492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:33.936764 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:33.936771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:33.937271 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:33.937272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:33.937436 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:33.938199 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:33.938211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:33.940503 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:33.943085 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:33.943102 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:33.943106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:33.943475 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:33.943942 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:33.943943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:33.943943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:33.944155 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:33.944675 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:33.944683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:33.944935 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 02:49:33.945081 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 02:49:33.945357 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260619 02:49:33.946524 30918 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46457
I20260619 02:49:33.946655 30918 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:33.946936 30918 heartbeater.cc:511] Master 127.26.98.62:46457 requested a full tablet report, sending...
I20260619 02:49:33.947450 30593 ts_manager.cc:194] Registered new tserver with Master: e9ba6ad20c2f49ed8f99d839bc6e4a8d (127.26.98.2:36693)
I20260619 02:49:33.947912 30593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:55997
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:33.976107 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:34.076344 30923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:34.076683 30923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:34.076730 30923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:34.078352 30923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:34.078423 30923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:34.078470 30923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:34.080493 30923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:34.080825 30923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:34.081100 30923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:34.086686 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:34.086874 30923 server_base.cc:1034] running on GCE node
W20260619 02:49:34.086686 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:34.086727 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:34.087250 30923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:34.087546 30923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:34.088717 30923 hybrid_clock.cc:648] HybridClock initialized: now 1781837374088701 us; error 37 us; skew 500 ppm
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837374, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:34.091679 30923 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:34.092260 30923 webserver.cc:460] Webserver started at http://127.26.98.3:45289/ using document root <none> and password file <none>
I20260619 02:49:34.092502 30923 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:34.092550 30923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:34.092770 30923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:34.094053 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "d94b618165734dad8ccb23966ad63a7e"
format_stamp: "Formatted at 2026-06-19 02:49:34 on dist-test-slave-0f9d"
server_key: "59ef8e1f65c793a2bfe19340241193e3"
server_key_iv: "13aef3ce472bc78b341e7b93e2c37dfb"
server_key_version: "encryptionkey@0"
I20260619 02:49:34.094455 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "d94b618165734dad8ccb23966ad63a7e"
format_stamp: "Formatted at 2026-06-19 02:49:34 on dist-test-slave-0f9d"
server_key: "59ef8e1f65c793a2bfe19340241193e3"
server_key_iv: "13aef3ce472bc78b341e7b93e2c37dfb"
server_key_version: "encryptionkey@0"
I20260619 02:49:34.096128 30923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 02:49:34.097169 30938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:34.097720 30923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:49:34.097851 30923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "d94b618165734dad8ccb23966ad63a7e"
format_stamp: "Formatted at 2026-06-19 02:49:34 on dist-test-slave-0f9d"
server_key: "59ef8e1f65c793a2bfe19340241193e3"
server_key_iv: "13aef3ce472bc78b341e7b93e2c37dfb"
server_key_version: "encryptionkey@0"
I20260619 02:49:34.097944 30923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:34.119853 30923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:34.122738 30923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:34.123008 30923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:34.123342 30923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:34.123771 30923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:34.123813 30923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:34.123850 30923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:34.123874 30923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:34.131747 30923 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:34355
I20260619 02:49:34.131834 31051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:34355 every 8 connection(s)
I20260619 02:49:34.132246 30923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:34.133288 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 30923
I20260619 02:49:34.133428 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:34.133837 27016 external_mini_cluster.cc:1428] Setting key 73c5a4354fedb98895cbb96a0e3bb9c9
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837374, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:34.146287 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:34.133526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:43785 (local address 127.26.98.62:46457)
0619 02:49:34.133597 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:34.133600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:34.134074 (+ 474us) server_negotiation.cc:407] Connection header received
0619 02:49:34.135762 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:34.135763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:34.135775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:34.135831 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:34.136716 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.137279 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.138670 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.138989 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.139062 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:34.142304 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:34.142307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:34.142307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:34.142317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:34.144435 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.145094 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.145095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.145095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.145166 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.145426 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.145427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.145427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.145560 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:34.145614 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:34.146035 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 02:49:34.146152 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:34.146533 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:34.133521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:46457 (local address 127.26.98.3:43785)
0619 02:49:34.133887 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:34.133953 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:34.135610 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:34.135941 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:34.135947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:34.136303 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:34.136576 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:34.136581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.137415 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.137417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:34.138506 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:34.138511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.139220 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.139223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:34.139431 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:34.140189 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:34.140204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:34.142150 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:34.144588 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:34.144603 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:34.144606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:34.144993 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:34.145229 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:34.145230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:34.145230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:34.145371 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:34.145615 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:34.145624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:34.145863 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 02:49:34.145983 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:49:34.146216 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
I20260619 02:49:34.147118 31052 heartbeater.cc:347] Connected to a master server at 127.26.98.62:46457
I20260619 02:49:34.147253 31052 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:34.147523 31052 heartbeater.cc:511] Master 127.26.98.62:46457 requested a full tablet report, sending...
I20260619 02:49:34.147989 30593 ts_manager.cc:194] Registered new tserver with Master: d94b618165734dad8ccb23966ad63a7e (127.26.98.3:34355)
I20260619 02:49:34.148456 30593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:43785
I20260619 02:49:34.158128 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[30548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:34.189738 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:34.178516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.26.98.1:40437)
0619 02:49:34.178798 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:34.178804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:34.178818 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:34.178850 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:34.178852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:34.178868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:34.178991 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:34.179771 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.180995 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.182167 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.182625 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.182718 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:34.185961 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:34.185977 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:34.185978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:34.185989 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:34.188137 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.188643 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.188647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.188647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.188719 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.188909 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.188910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.188911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.189165 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:34.189422 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:34.189476 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:49:34.189578 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":157,"threads_started":1}
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[30548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:34.200553 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:34.191566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.26.98.62:46457)
0619 02:49:34.191625 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:34.191627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:34.191637 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:34.191728 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:34.191730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:34.191741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:34.191790 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:34.192347 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.192904 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.193731 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:34.193942 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:34.194026 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:34.196981 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:34.196984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:34.196984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:34.196996 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:34.198691 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.199329 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.199332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.199332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.199457 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:34.199770 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:34.199771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:34.199772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:34.200046 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:34.200154 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:34.200349 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:49:34.200448 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 02:49:34.200958 30593 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51586
I20260619 02:49:34.201583 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30655
I20260619 02:49:34.207178 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30789
I20260619 02:49:34.211751 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30923
I20260619 02:49:34.216871 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 30561
2026-06-19T02:49:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2002 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[31064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[31064](info): set up 2 sockets
Jun 19 02:49:34 dist-test-slave-0f9d krb5kdc[31064](info): commencing operation
krb5kdc: starting...
W20260619 02:49:35.215386 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:35.355973 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36003
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:35.448884 31077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:35.449193 31077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:35.449223 31077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:35.450778 31077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:35.450866 31077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:35.450923 31077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:35.450937 31077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:35.450949 31077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:35.453179 31077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:35.453553 31077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:35.453868 31077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:35.459880 31082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.459956 31083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.460216 31077 server_base.cc:1034] running on GCE node
W20260619 02:49:35.460379 31085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.460870 31077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:35.461418 31077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:35.462630 31077 hybrid_clock.cc:648] HybridClock initialized: now 1781837375462580 us; error 71 us; skew 500 ppm
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:35.473791 31077 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:35.474349 31077 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:35.474423 31077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:35.474553 31077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:35.476008 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "a93063c9819c41a1b5b5b7dc8993fb0f"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "862035f897b83c2d376a95a312b221a8"
server_key_iv: "a28994bdcc9ce6981e6c2cc44ce911fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.476413 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "a93063c9819c41a1b5b5b7dc8993fb0f"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "862035f897b83c2d376a95a312b221a8"
server_key_iv: "a28994bdcc9ce6981e6c2cc44ce911fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.478236 31077 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:35.479444 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.479763 31077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:35.479898 31077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "a93063c9819c41a1b5b5b7dc8993fb0f"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "862035f897b83c2d376a95a312b221a8"
server_key_iv: "a28994bdcc9ce6981e6c2cc44ce911fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.480005 31077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:35.504706 31077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:35.507340 31077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:35.507506 31077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:35.512459 31077 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36003
I20260619 02:49:35.512501 31143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36003 every 8 connection(s)
I20260619 02:49:35.512867 31077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:35.513990 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:35.517036 31144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: Bootstrap starting.
I20260619 02:49:35.517745 31144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:35.518070 31144 log.cc:824] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:35.518944 31144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: No bootstrap required, opened a new log
I20260619 02:49:35.520814 31144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } }
I20260619 02:49:35.521001 31144 raft_consensus.cc:374] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:35.521026 31144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93063c9819c41a1b5b5b7dc8993fb0f, State: Initialized, Role: FOLLOWER
I20260619 02:49:35.521189 31144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } }
I20260619 02:49:35.521267 31144 raft_consensus.cc:388] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:35.521296 31144 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:35.521337 31144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:35.522374 31144 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } }
I20260619 02:49:35.522485 31144 leader_election.cc:302] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93063c9819c41a1b5b5b7dc8993fb0f; no voters:
I20260619 02:49:35.522703 31144 leader_election.cc:288] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:35.522805 31147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:35.523023 31147 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [term 1 LEADER]: Becoming Leader. State: Replica: a93063c9819c41a1b5b5b7dc8993fb0f, State: Running, Role: LEADER
I20260619 02:49:35.523034 31144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:35.523030 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31077
I20260619 02:49:35.523172 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:35.523206 31147 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } }
I20260619 02:49:35.523380 27016 external_mini_cluster.cc:1428] Setting key ac0a1fd2bd9216071d40bf8938980b82
I20260619 02:49:35.523772 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } } }
I20260619 02:49:35.523869 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:35.524099 31156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:35.524609 31149 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93063c9819c41a1b5b5b7dc8993fb0f. Latest consensus state: current_term: 1 leader_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93063c9819c41a1b5b5b7dc8993fb0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36003 } } }
I20260619 02:49:35.524689 31149 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:35.524966 31156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:35.526995 31156 catalog_manager.cc:1347] Generated new cluster ID: bb3efc99ea54422d97dea741c5da5f6d
I20260619 02:49:35.527067 31156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:35.537194 31156 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:35.537820 31156 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:35.539135 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.524161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.26.98.62:36003)
0619 02:49:35.524373 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:35.524386 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:35.524903 (+ 517us) server_negotiation.cc:407] Connection header received
0619 02:49:35.525113 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:35.525123 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:35.525194 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:35.525364 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:35.526315 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.527841 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.529019 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.529747 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.529887 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.533681 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:35.533691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:35.533692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:35.533714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:35.536268 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.536818 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.536822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.536823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.536949 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.537231 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.537233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.537233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.537628 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:35.537826 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:35.538528 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 02:49:35.538778 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260619 02:49:35.541612 31156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: Generated new TSK 0
I20260619 02:49:35.541896 31156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:35.576191 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:35.676733 31170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:35.677035 31170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:35.677066 31170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:35.678647 31170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:35.678737 31170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:35.678776 31170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:35.680979 31170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:35.681352 31170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:35.681645 31170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:35.688211 31176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.688263 31170 server_base.cc:1034] running on GCE node
W20260619 02:49:35.688350 31178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.688241 31175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.688828 31170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:35.689502 31170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:35.690719 31170 hybrid_clock.cc:648] HybridClock initialized: now 1781837375690687 us; error 52 us; skew 500 ppm
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:35.694430 31170 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:35.694958 31170 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:35.695058 31170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:35.695235 31170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:35.696568 31170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "cded439e57e04402911bb046f73ef46a"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "c02593b19dc67737514588a3c5247bd2"
server_key_iv: "e84cb7a7a8009e4c5b6e5fa196af2387"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.697017 31170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "cded439e57e04402911bb046f73ef46a"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "c02593b19dc67737514588a3c5247bd2"
server_key_iv: "e84cb7a7a8009e4c5b6e5fa196af2387"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.698661 31170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:35.699761 31184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.699997 31170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:35.700107 31170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "cded439e57e04402911bb046f73ef46a"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "c02593b19dc67737514588a3c5247bd2"
server_key_iv: "e84cb7a7a8009e4c5b6e5fa196af2387"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.700212 31170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:35.713908 31170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:35.716645 31170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:35.716809 31170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:35.717088 31170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:35.717460 31170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:35.717496 31170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.717533 31170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:35.717556 31170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.725751 31170 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:43521
I20260619 02:49:35.725831 31297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:43521 every 8 connection(s)
I20260619 02:49:35.726179 31170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:35.733007 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31170
I20260619 02:49:35.733170 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:35.733387 27016 external_mini_cluster.cc:1428] Setting key ea0fb99bb7ec5d1d7b6fa289ef0e51f8
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:35.739462 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.727196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:59011 (local address 127.26.98.62:36003)
0619 02:49:35.727274 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:35.727276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:35.727764 (+ 488us) server_negotiation.cc:407] Connection header received
0619 02:49:35.728535 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:35.728537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:35.728553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:35.728638 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:35.730021 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.730675 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.732006 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.732268 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.732343 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.735636 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:35.735638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:35.735638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:35.735649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:35.737331 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.738152 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.738183 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.738183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.738261 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.738564 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.738565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.738565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.738707 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:35.738795 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:35.739248 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 02:49:35.739375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:35.739706 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.727170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36003 (local address 127.26.98.1:59011)
0619 02:49:35.727555 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:35.727618 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:35.728354 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:35.728808 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:35.728813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:35.729225 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:35.729876 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:35.729884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.730799 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.730800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:35.731883 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:35.731887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.732418 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.732419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:35.732585 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.733378 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:35.733389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:35.735501 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:35.737619 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:35.737635 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:35.737640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:35.738028 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:35.738331 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:35.738332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:35.738333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:35.738491 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:35.738770 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:35.738778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:35.739038 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 02:49:35.739187 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 02:49:35.739413 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":92,"threads_started":1}
I20260619 02:49:35.740319 31298 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36003
I20260619 02:49:35.740460 31298 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:35.740666 31298 heartbeater.cc:511] Master 127.26.98.62:36003 requested a full tablet report, sending...
I20260619 02:49:35.741308 31108 ts_manager.cc:194] Registered new tserver with Master: cded439e57e04402911bb046f73ef46a (127.26.98.1:43521)
I20260619 02:49:35.742012 31108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:59011
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:35.763898 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:35.864344 31303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:35.864655 31303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:35.864687 31303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:35.866604 31303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:35.866707 31303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:35.866753 31303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:35.869673 31303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:35.870066 31303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:35.870368 31303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:35.876525 31311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.876525 31308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.876530 31309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.876760 31303 server_base.cc:1034] running on GCE node
I20260619 02:49:35.877092 31303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:35.877460 31303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:35.878654 31303 hybrid_clock.cc:648] HybridClock initialized: now 1781837375878622 us; error 53 us; skew 500 ppm
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:35.886121 31303 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:35.886672 31303 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:35.886742 31303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:35.886962 31303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:35.888413 31303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "51a78014b66c496e9346a4d04dda447e"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "02d133137df43765e77ba55219fb98db"
server_key_iv: "1026a5f12dbd3e0e6b24286d24981d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.888868 31303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "51a78014b66c496e9346a4d04dda447e"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "02d133137df43765e77ba55219fb98db"
server_key_iv: "1026a5f12dbd3e0e6b24286d24981d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.890554 31303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:35.891556 31317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.891748 31303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:35.891840 31303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "51a78014b66c496e9346a4d04dda447e"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-0f9d"
server_key: "02d133137df43765e77ba55219fb98db"
server_key_iv: "1026a5f12dbd3e0e6b24286d24981d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.891928 31303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:35.916687 31303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:35.919631 31303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:35.919816 31303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:35.920118 31303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:35.920495 31303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:35.920529 31303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.920564 31303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:35.920584 31303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.927737 31303 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:44571
I20260619 02:49:35.927778 31430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:44571 every 8 connection(s)
I20260619 02:49:35.928121 31303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:35.931249 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31303
I20260619 02:49:35.931365 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:35.931564 27016 external_mini_cluster.cc:1428] Setting key 28fb193957de1d4fcd518f7833d1b2f1
Jun 19 02:49:35 dist-test-slave-0f9d krb5kdc[31064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:35.941504 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.929000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:40005 (local address 127.26.98.62:36003)
0619 02:49:35.929085 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:35.929087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:35.929343 (+ 256us) server_negotiation.cc:407] Connection header received
0619 02:49:35.929951 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:35.929953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:35.929968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:35.930026 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:35.931284 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.931972 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.933823 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.934062 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.934146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.937427 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:35.937431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:35.937432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:35.937446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:35.939606 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.940238 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.940239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.940240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.940320 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.940645 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.940646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.940646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.940844 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:35.940916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:35.941263 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 02:49:35.941391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:49:35.941741 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.928995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36003 (local address 127.26.98.2:40005)
0619 02:49:35.929189 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:35.929235 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:35.929836 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:35.930124 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:35.930129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:35.930452 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:35.931110 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:35.931118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.932148 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.932150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:35.933635 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:35.933640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.934167 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.934168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:35.934378 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.935187 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:35.935200 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:35.937221 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:35.939770 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:35.939789 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:35.939794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:35.940130 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:35.940389 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:35.940391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:35.940391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:35.940569 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:35.940915 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:35.940923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:35.941115 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 02:49:35.941229 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 02:49:35.941461 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260619 02:49:35.942708 31431 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36003
I20260619 02:49:35.942858 31431 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:35.943286 31431 heartbeater.cc:511] Master 127.26.98.62:36003 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:49:35.943933 31108 ts_manager.cc:194] Registered new tserver with Master: 51a78014b66c496e9346a4d04dda447e (127.26.98.2:44571)
I20260619 02:49:35.944700 31108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:40005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:35.966560 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:36.073611 31436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:36.073879 31436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:36.073910 31436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:36.075399 31436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:36.075467 31436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:36.075505 31436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:36.077422 31436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:36.077759 31436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:36.078037 31436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:36.083757 31444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.083918 31442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.083757 31441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:36.084039 31436 server_base.cc:1034] running on GCE node
I20260619 02:49:36.084344 31436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:36.084671 31436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:36.085882 31436 hybrid_clock.cc:648] HybridClock initialized: now 1781837376085867 us; error 37 us; skew 500 ppm
Jun 19 02:49:36 dist-test-slave-0f9d krb5kdc[31064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837376, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:36.089139 31436 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:36.089639 31436 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:36.089708 31436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:36.089864 31436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:36.091355 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "8893e45f6875444b907600a40afebfc7"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "ceb937e6be18dab747d00c144612df53"
server_key_iv: "828610bbfad08409fecc9f7725efbff2"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.091825 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "8893e45f6875444b907600a40afebfc7"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "ceb937e6be18dab747d00c144612df53"
server_key_iv: "828610bbfad08409fecc9f7725efbff2"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.093395 31436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:36.094362 31450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.094807 31436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:36.094970 31436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "8893e45f6875444b907600a40afebfc7"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "ceb937e6be18dab747d00c144612df53"
server_key_iv: "828610bbfad08409fecc9f7725efbff2"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.095068 31436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:36.123802 31436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:36.126909 31436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:36.127103 31436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:36.127439 31436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:36.127844 31436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:36.127883 31436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.127916 31436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:36.127939 31436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.136224 31436 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:44523
I20260619 02:49:36.136337 31563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:44523 every 8 connection(s)
I20260619 02:49:36.136675 31436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:36.144377 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31436
I20260619 02:49:36.144493 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:36.144703 27016 external_mini_cluster.cc:1428] Setting key e4931dcc9432f09d6dfa263e6c38f579
Jun 19 02:49:36 dist-test-slave-0f9d krb5kdc[31064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837376, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:36.149456 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.137696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59083 (local address 127.26.98.62:36003)
0619 02:49:36.137813 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.137815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.138051 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:49:36.138764 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.138766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.138783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.138944 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:36.140564 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.141140 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.142690 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.142993 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.143070 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.146318 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:36.146319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:36.146320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:36.146330 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:36.147763 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:36.148304 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:36.148305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:36.148306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:36.148377 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:36.148645 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:36.148645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:36.148646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:36.148804 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:36.148872 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.149212 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.149345 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:49:36.149679 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.137623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36003 (local address 127.26.98.3:59083)
0619 02:49:36.137836 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.137890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.138520 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.139220 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.139225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.139680 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:36.140355 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.140365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.141405 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.141408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.142506 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.142510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.143231 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.143236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.143415 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.144214 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:36.144227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:36.146211 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:36.147872 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:36.147890 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:36.147894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:36.148220 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:36.148439 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:36.148440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:36.148440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:36.148583 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:36.148870 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:36.148878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:36.149068 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.149179 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.149398 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260619 02:49:36.150238 31564 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36003
I20260619 02:49:36.150355 31564 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:36.150518 31564 heartbeater.cc:511] Master 127.26.98.62:36003 requested a full tablet report, sending...
I20260619 02:49:36.151222 31108 ts_manager.cc:194] Registered new tserver with Master: 8893e45f6875444b907600a40afebfc7 (127.26.98.3:44523)
I20260619 02:49:36.151734 31108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:59083
I20260619 02:49:36.157696 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:36.167256 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.159520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.26.98.62:36003)
0619 02:49:36.159592 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.159595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.159804 (+ 209us) server_negotiation.cc:407] Connection header received
0619 02:49:36.159850 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.159851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.159863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.159934 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:36.160609 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.161198 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.162038 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.162280 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.162347 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.163904 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:36.163906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:36.163906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:36.163916 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:36.165431 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:36.165883 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:36.165885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:36.165885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:36.165959 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:36.166377 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:36.166378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:36.166379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:36.166601 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:36.166710 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.167008 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.167146 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:36.169668 31108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:36.171868 31108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:36.181044 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.175744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:43521 (local address 127.0.0.1:42200)
0619 02:49:36.176234 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.176243 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.176290 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.176731 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.176732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.176735 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:36.176788 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.176790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.177757 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.177758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.179396 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.179397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180373 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.180502 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.180521 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.180607 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.180910 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":57,"thread_start_us":109,"threads_started":1}
I20260619 02:49:36.181092 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.176231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.26.98.1:43521)
0619 02:49:36.176473 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.176478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.176488 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:36.176533 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.176535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.176551 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.176631 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:36.176871 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.177627 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.179549 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180215 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180324 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.180409 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.180673 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.180961 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260619 02:49:36.181551 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.175663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44571 (local address 127.0.0.1:45936)
0619 02:49:36.175957 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.176022 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.176120 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.176551 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.176555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.176557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:36.176657 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.176661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.178153 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.178154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.179332 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.179337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180879 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.180998 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.181001 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.181383 (+ 382us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.181441 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":41,"spinlock_wait_cycles":896,"thread_start_us":140,"threads_started":1}
I20260619 02:49:36.181871 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.175953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.26.98.2:44571)
0619 02:49:36.176166 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.176174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.176250 (+ 76us) server_negotiation.cc:407] Connection header received
0619 02:49:36.176314 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.176317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.176338 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.176426 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:36.176742 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.177975 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.179441 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180612 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180734 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.180816 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.181710 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.181780 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
I20260619 02:49:36.182013 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.175962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.26.98.3:44523)
0619 02:49:36.176177 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.176180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.176382 (+ 202us) server_negotiation.cc:407] Connection header received
0619 02:49:36.176428 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.176430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.176447 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.176519 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:36.176755 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.177973 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.179528 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180610 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180744 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.180816 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.181764 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.181834 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260619 02:49:36.182245 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.175663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:44523 (local address 127.0.0.1:47632)
0619 02:49:36.176232 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.176244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.176321 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.176617 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.176617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.176619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:36.176658 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.176662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.178117 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.178119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.179388 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.179397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.180880 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.180881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.180993 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.180999 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.181082 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.182179 (+ 1097us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":34,"thread_start_us":114,"threads_started":1}
I20260619 02:49:36.182181 31217 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed4b57035094b62b92d975b8e57be87 (DEFAULT_TABLE table=test-table [id=892acadaae244e30bb795ac58599283c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:36.182615 31217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed4b57035094b62b92d975b8e57be87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:36.183372 31498 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed4b57035094b62b92d975b8e57be87 (DEFAULT_TABLE table=test-table [id=892acadaae244e30bb795ac58599283c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:36.183874 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed4b57035094b62b92d975b8e57be87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:36.187132 31581 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Bootstrap starting.
I20260619 02:49:36.187502 31582 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Bootstrap starting.
I20260619 02:49:36.188124 31581 tablet_bootstrap.cc:654] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:36.188339 31582 tablet_bootstrap.cc:654] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:36.188231 31365 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed4b57035094b62b92d975b8e57be87 (DEFAULT_TABLE table=test-table [id=892acadaae244e30bb795ac58599283c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:36.188700 31365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed4b57035094b62b92d975b8e57be87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:36.188843 31581 log.cc:824] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:36.188939 31582 log.cc:824] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:36.189749 31581 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: No bootstrap required, opened a new log
I20260619 02:49:36.189815 31581 ts_tablet_manager.cc:1397] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:36.191510 31582 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: No bootstrap required, opened a new log
I20260619 02:49:36.191614 31582 ts_tablet_manager.cc:1397] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:49:36.191644 31581 raft_consensus.cc:348] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.191833 31581 raft_consensus.cc:374] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:36.191875 31581 raft_consensus.cc:729] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cded439e57e04402911bb046f73ef46a, State: Initialized, Role: FOLLOWER
I20260619 02:49:36.192014 31581 consensus_queue.cc:260] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.192390 31581 ts_tablet_manager.cc:1428] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:36.192399 31298 heartbeater.cc:503] Master 127.26.98.62:36003 was elected leader, sending a full tablet report...
I20260619 02:49:36.193493 31586 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Bootstrap starting.
I20260619 02:49:36.193883 31582 raft_consensus.cc:348] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.194087 31582 raft_consensus.cc:374] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:36.194134 31582 raft_consensus.cc:729] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8893e45f6875444b907600a40afebfc7, State: Initialized, Role: FOLLOWER
I20260619 02:49:36.194270 31582 consensus_queue.cc:260] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.194344 31586 tablet_bootstrap.cc:654] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:36.194631 31564 heartbeater.cc:503] Master 127.26.98.62:36003 was elected leader, sending a full tablet report...
I20260619 02:49:36.194639 31582 ts_tablet_manager.cc:1428] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:36.195040 31586 log.cc:824] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:36.196043 31586 tablet_bootstrap.cc:492] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: No bootstrap required, opened a new log
I20260619 02:49:36.196151 31586 ts_tablet_manager.cc:1397] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:36.198460 31586 raft_consensus.cc:348] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.198984 31586 raft_consensus.cc:374] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:36.199044 31586 raft_consensus.cc:729] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a78014b66c496e9346a4d04dda447e, State: Initialized, Role: FOLLOWER
I20260619 02:49:36.199229 31586 consensus_queue.cc:260] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.199748 31586 ts_tablet_manager.cc:1428] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:36.199759 31431 heartbeater.cc:503] Master 127.26.98.62:36003 was elected leader, sending a full tablet report...
W20260619 02:49:36.227057 31299 tablet.cc:2369] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:36.387538 31565 tablet.cc:2369] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:36.429339 31432 tablet.cc:2369] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:36.510056 31585 raft_consensus.cc:482] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:36.510288 31585 raft_consensus.cc:504] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.510749 31585 leader_election.cc:288] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a78014b66c496e9346a4d04dda447e (127.26.98.2:44571), 8893e45f6875444b907600a40afebfc7 (127.26.98.3:44523)
I20260619 02:49:36.514402 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.511010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:44571 (local address 127.26.98.1:57631)
0619 02:49:36.511195 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.511208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.511294 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.511441 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.511442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.511444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:36.511518 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.511521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.512281 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.512283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.513291 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.513296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.514022 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.514025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.514128 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.514133 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.514209 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.514273 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260619 02:49:36.514387 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.511091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:57631 (local address 127.26.98.2:44571)
0619 02:49:36.511154 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.511157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.511295 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:49:36.511359 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.511360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.511374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.511429 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:36.511589 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.512137 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.513421 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.513898 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.514007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.514023 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.514225 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.514293 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:36.514925 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.511028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:44523 (local address 127.26.98.1:35483)
0619 02:49:36.511205 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.511214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.511294 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.511440 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.511441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.511443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:36.511559 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.511561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.512470 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.512472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.513630 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:36.513634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.514585 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.514586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:36.514668 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.514674 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.514731 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.514795 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":24,"thread_start_us":104,"threads_started":1}
I20260619 02:49:36.514925 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.511147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35483 (local address 127.26.98.3:44523)
0619 02:49:36.511200 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.511202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.511285 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:49:36.511359 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.511360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.511372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.511428 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:36.511727 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.512378 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.513831 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.514432 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.514521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.514535 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.514756 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.514809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:49:36.514847 31385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed4b57035094b62b92d975b8e57be87" candidate_uuid: "cded439e57e04402911bb046f73ef46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a78014b66c496e9346a4d04dda447e" is_pre_election: true
I20260619 02:49:36.515125 31385 raft_consensus.cc:2454] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cded439e57e04402911bb046f73ef46a in term 0.
I20260619 02:49:36.515455 31188 leader_election.cc:302] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a78014b66c496e9346a4d04dda447e, cded439e57e04402911bb046f73ef46a; no voters:
I20260619 02:49:36.515434 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed4b57035094b62b92d975b8e57be87" candidate_uuid: "cded439e57e04402911bb046f73ef46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8893e45f6875444b907600a40afebfc7" is_pre_election: true
I20260619 02:49:36.515653 31585 raft_consensus.cc:2781] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:36.515700 31518 raft_consensus.cc:2454] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cded439e57e04402911bb046f73ef46a in term 0.
I20260619 02:49:36.515729 31585 raft_consensus.cc:482] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:36.515755 31585 raft_consensus.cc:3037] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:36.516645 31585 raft_consensus.cc:504] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.516831 31585 leader_election.cc:288] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [CANDIDATE]: Term 1 election: Requested vote from peers 51a78014b66c496e9346a4d04dda447e (127.26.98.2:44571), 8893e45f6875444b907600a40afebfc7 (127.26.98.3:44523)
I20260619 02:49:36.517143 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed4b57035094b62b92d975b8e57be87" candidate_uuid: "cded439e57e04402911bb046f73ef46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8893e45f6875444b907600a40afebfc7"
I20260619 02:49:36.517213 31385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed4b57035094b62b92d975b8e57be87" candidate_uuid: "cded439e57e04402911bb046f73ef46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a78014b66c496e9346a4d04dda447e"
I20260619 02:49:36.517319 31385 raft_consensus.cc:3037] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:36.517319 31518 raft_consensus.cc:3037] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:36.518661 31385 raft_consensus.cc:2454] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cded439e57e04402911bb046f73ef46a in term 1.
I20260619 02:49:36.518666 31518 raft_consensus.cc:2454] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cded439e57e04402911bb046f73ef46a in term 1.
I20260619 02:49:36.518985 31188 leader_election.cc:302] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a78014b66c496e9346a4d04dda447e, cded439e57e04402911bb046f73ef46a; no voters:
I20260619 02:49:36.519135 31585 raft_consensus.cc:2781] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:36.519344 31585 raft_consensus.cc:686] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 1 LEADER]: Becoming Leader. State: Replica: cded439e57e04402911bb046f73ef46a, State: Running, Role: LEADER
I20260619 02:49:36.519531 31585 consensus_queue.cc:237] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } }
I20260619 02:49:36.520674 31107 catalog_manager.cc:5495] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a reported cstate change: term changed from 0 to 1, leader changed from <none> to cded439e57e04402911bb046f73ef46a (127.26.98.1). New cstate: current_term: 1 leader_uuid: "cded439e57e04402911bb046f73ef46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cded439e57e04402911bb046f73ef46a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 43521 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:36.541716 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.538684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.26.98.1:43521)
0619 02:49:36.538771 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.538773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.538805 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:36.538850 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.538850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.538875 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.539000 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:36.539335 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.539886 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.540806 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:36.541015 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:36.541074 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:36.541296 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:36.541440 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:36.541521 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.541592 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.541617 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:49:36.544389 31385 raft_consensus.cc:1261] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 1 FOLLOWER]: Refusing update from remote peer cded439e57e04402911bb046f73ef46a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:36.544389 31518 raft_consensus.cc:1261] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 1 FOLLOWER]: Refusing update from remote peer cded439e57e04402911bb046f73ef46a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:36.544993 31585 consensus_queue.cc:1035] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a78014b66c496e9346a4d04dda447e" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:36.545255 31592 consensus_queue.cc:1035] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8893e45f6875444b907600a40afebfc7" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:36.549435 31597 mvcc.cc:205] Tried to move back new op lower bound from 7298405894321471488 to 7298405894224941056. Current Snapshot: MvccSnapshot[applied={T|T < 7298405894321471488}]
I20260619 02:49:36.549600 31595 mvcc.cc:205] Tried to move back new op lower bound from 7298405894321471488 to 7298405894224941056. Current Snapshot: MvccSnapshot[applied={T|T < 7298405894321471488 or (T in {7298405894321471488})}]
I20260619 02:49:36.551542 31107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:36.551695 31107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:36.552789 31107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a93063c9819c41a1b5b5b7dc8993fb0f: Sending DeleteTablet for 3 replicas of tablet 1ed4b57035094b62b92d975b8e57be87
I20260619 02:49:36.553323 31365 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed4b57035094b62b92d975b8e57be87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:36 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260619 02:49:36.553432 31498 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed4b57035094b62b92d975b8e57be87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:36 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47632
I20260619 02:49:36.553828 31606 tablet_replica.cc:330] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: stopping tablet replica
I20260619 02:49:36.553825 31217 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed4b57035094b62b92d975b8e57be87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:36 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:42200
I20260619 02:49:36.554001 31606 raft_consensus.cc:2229] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:36.554142 31607 tablet_replica.cc:330] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a: stopping tablet replica
I20260619 02:49:36.554169 31606 raft_consensus.cc:2258] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:36.554337 31607 raft_consensus.cc:2229] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:36.554437 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31170
I20260619 02:49:36.554569 31607 raft_consensus.cc:2258] T 1ed4b57035094b62b92d975b8e57be87 P cded439e57e04402911bb046f73ef46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:36.554768 31606 ts_tablet_manager.cc:1905] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:36.556178 31605 tablet_replica.cc:330] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: stopping tablet replica
I20260619 02:49:36.556363 31605 raft_consensus.cc:2229] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:36.556531 31605 raft_consensus.cc:2258] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:36.557783 31606 ts_tablet_manager.cc:1918] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:36.557875 31606 log.cc:1196] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/1ed4b57035094b62b92d975b8e57be87
I20260619 02:49:36.558131 31606 ts_tablet_manager.cc:1939] T 1ed4b57035094b62b92d975b8e57be87 P 8893e45f6875444b907600a40afebfc7: Deleting consensus metadata
I20260619 02:49:36.558710 31093 catalog_manager.cc:4841] TS 8893e45f6875444b907600a40afebfc7 (127.26.98.3:44523): tablet 1ed4b57035094b62b92d975b8e57be87 (table test-table [id=892acadaae244e30bb795ac58599283c]) successfully deleted
I20260619 02:49:36.559331 31605 ts_tablet_manager.cc:1905] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:36.561205 31605 ts_tablet_manager.cc:1918] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:36.561296 31605 log.cc:1196] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/1ed4b57035094b62b92d975b8e57be87
W20260619 02:49:36.561301 31092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:43521 (error 108)
I20260619 02:49:36.561508 31605 ts_tablet_manager.cc:1939] T 1ed4b57035094b62b92d975b8e57be87 P 51a78014b66c496e9346a4d04dda447e: Deleting consensus metadata
I20260619 02:49:36.562373 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31303
I20260619 02:49:36.562371 31095 catalog_manager.cc:4841] TS 51a78014b66c496e9346a4d04dda447e (127.26.98.2:44571): tablet 1ed4b57035094b62b92d975b8e57be87 (table test-table [id=892acadaae244e30bb795ac58599283c]) successfully deleted
I20260619 02:49:36.562527 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.562382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:43521 (local address 127.0.0.1:42220)
0619 02:49:36.562444 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.562491 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:43521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 02:49:36.563331 31092 catalog_manager.cc:4584] TS cded439e57e04402911bb046f73ef46a (127.26.98.1:43521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ed4b57035094b62b92d975b8e57be87: Network error: Client connection negotiation failed: client connection to 127.26.98.1:43521: connect: Connection refused (error 111)
I20260619 02:49:36.567937 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31436
I20260619 02:49:36.573069 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31077
2026-06-19T02:49:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2356 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:36Z Disabled control of system clock
I20260619 02:49:36.603780 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44455
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:36.703295 31612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:36.703631 31612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:36.703660 31612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:36.705289 31612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:36.705384 31612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:36.705397 31612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:36.705410 31612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:36.705421 31612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:36.707361 31612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:36.707744 31612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:36.708114 31612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:36.714416 31618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.714447 31620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.714586 31617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:36.714923 31612 server_base.cc:1034] running on GCE node
I20260619 02:49:36.715322 31612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:36.715716 31612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:36.716933 31612 hybrid_clock.cc:648] HybridClock initialized: now 1781837376716911 us; error 46 us; skew 500 ppm
I20260619 02:49:36.718689 31612 webserver.cc:460] Webserver started at http://127.26.98.62:46285/ using document root <none> and password file <none>
I20260619 02:49:36.719054 31612 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:36.719121 31612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:36.719291 31612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:36.720661 31612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "bd9554b839434cdd9256fd8a3cfe748c"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "881598fddf5f2e2f6ee98438f23a7c25"
server_key_iv: "deccfa268c57fc509ba1a887aea14764"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.721556 31612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "bd9554b839434cdd9256fd8a3cfe748c"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "881598fddf5f2e2f6ee98438f23a7c25"
server_key_iv: "deccfa268c57fc509ba1a887aea14764"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.724193 31612 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260619 02:49:36.725421 31626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.725824 31612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:36.726157 31612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "bd9554b839434cdd9256fd8a3cfe748c"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "881598fddf5f2e2f6ee98438f23a7c25"
server_key_iv: "deccfa268c57fc509ba1a887aea14764"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.726282 31612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:36.730576 31612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:36.731004 31612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:36.731154 31612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:36.736431 31612 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:44455
I20260619 02:49:36.736524 31678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:44455 every 8 connection(s)
I20260619 02:49:36.737002 31612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:36.738194 31679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:36.740401 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31612
I20260619 02:49:36.740509 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:36.740690 27016 external_mini_cluster.cc:1428] Setting key a23fb2d7f575040544c3ae12d810560f
I20260619 02:49:36.742272 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: Bootstrap starting.
I20260619 02:49:36.743172 31679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:36.743580 31679 log.cc:824] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:36.744136 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.741388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.26.98.62:44455)
0619 02:49:36.741645 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.741656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.741686 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:49:36.742178 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.742187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.742245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.742410 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:36.743544 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:36.743549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:36.743550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:36.743555 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:36.743618 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:36.743637 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:36.743722 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.743873 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.743948 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":108,"threads_started":1}
I20260619 02:49:36.744552 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: No bootstrap required, opened a new log
I20260619 02:49:36.746423 31679 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } }
I20260619 02:49:36.746665 31679 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:36.746690 31679 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9554b839434cdd9256fd8a3cfe748c, State: Initialized, Role: FOLLOWER
I20260619 02:49:36.746820 31679 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } }
I20260619 02:49:36.746959 31679 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:36.746997 31679 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:36.747042 31679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:36.748379 31679 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } }
I20260619 02:49:36.748540 31679 leader_election.cc:302] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9554b839434cdd9256fd8a3cfe748c; no voters:
I20260619 02:49:36.748800 31679 leader_election.cc:288] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:36.748865 31684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:36.748994 31684 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [term 1 LEADER]: Becoming Leader. State: Replica: bd9554b839434cdd9256fd8a3cfe748c, State: Running, Role: LEADER
I20260619 02:49:36.749105 31684 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } }
I20260619 02:49:36.749181 31679 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:36.749634 31685 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9554b839434cdd9256fd8a3cfe748c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } } }
I20260619 02:49:36.749660 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9554b839434cdd9256fd8a3cfe748c. Latest consensus state: current_term: 1 leader_uuid: "bd9554b839434cdd9256fd8a3cfe748c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9554b839434cdd9256fd8a3cfe748c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44455 } } }
I20260619 02:49:36.749749 31685 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:36.749754 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:36.750041 31690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:36.750612 31690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:36.752447 31690 catalog_manager.cc:1347] Generated new cluster ID: 750c11730d5e40f794d5b361ae24b3af
I20260619 02:49:36.752511 31690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:36.771494 31690 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:36.771775 31690 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:36.780364 31690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: Generated new TSK 0
I20260619 02:49:36.780668 31690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:36.785671 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--builtin_ntp_servers=127.26.98.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:36.885629 31703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:36.885975 31703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:36.886018 31703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:36.887954 31703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:36.888058 31703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:36.888111 31703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:36.890092 31703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:36.890475 31703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:36.890862 31703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:36.896952 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.896942 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:36.896942 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:36.897501 31703 server_base.cc:1034] running on GCE node
I20260619 02:49:36.897786 31703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:36.898111 31703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:36.899348 31703 hybrid_clock.cc:648] HybridClock initialized: now 1781837376899320 us; error 39 us; skew 500 ppm
I20260619 02:49:36.901095 31703 webserver.cc:460] Webserver started at http://127.26.98.1:35507/ using document root <none> and password file <none>
I20260619 02:49:36.901402 31703 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:36.901461 31703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:36.901605 31703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:36.902984 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "a397df1bc5174ee792a1b9505414365d"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "f8eee301acf56f8687dd07e96cf0e2aa"
server_key_iv: "c1c399f45eb63365d78f38d37dd8b7b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.903406 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "a397df1bc5174ee792a1b9505414365d"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "f8eee301acf56f8687dd07e96cf0e2aa"
server_key_iv: "c1c399f45eb63365d78f38d37dd8b7b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.905069 31703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:49:36.906118 31717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.906473 31703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:36.906579 31703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "a397df1bc5174ee792a1b9505414365d"
format_stamp: "Formatted at 2026-06-19 02:49:36 on dist-test-slave-0f9d"
server_key: "f8eee301acf56f8687dd07e96cf0e2aa"
server_key_iv: "c1c399f45eb63365d78f38d37dd8b7b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:36.906687 31703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:36.911090 31703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:36.911397 31703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:36.911536 31703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:36.911755 31703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:36.912339 31703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:36.912405 31703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.912432 31703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:36.912444 31703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:36.919952 31703 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:45941
I20260619 02:49:36.920068 31830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:45941 every 8 connection(s)
I20260619 02:49:36.920665 31703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:36.922155 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31703
I20260619 02:49:36.922314 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:36.922550 27016 external_mini_cluster.cc:1428] Setting key d2c4c92b86df45acadf72dc346dac880
I20260619 02:49:36.923770 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--builtin_ntp_servers=127.26.98.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:49:36.924930 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.921806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:34547 (local address 127.26.98.62:44455)
0619 02:49:36.921952 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:36.921955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:36.922409 (+ 454us) server_negotiation.cc:407] Connection header received
0619 02:49:36.923085 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:36.923086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:36.923094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:36.923149 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:36.924663 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:36.924665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:36.924666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:36.924671 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:36.924692 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:36.924696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:36.924754 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:36.924849 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:49:36.924865 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 02:49:36.925206 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:36.921728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44455 (local address 127.26.98.1:34547)
0619 02:49:36.922127 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:36.922191 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:36.922983 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:36.923228 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:36.923233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:36.923243 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:36.924421 (+ 1178us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:36.924422 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:36.924478 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:36.924529 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:36.924767 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:36.924773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:36.924777 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:36.924857 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:49:36.924939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":87,"threads_started":1}
I20260619 02:49:36.925818 31831 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44455
I20260619 02:49:36.925921 31831 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:36.926085 31831 heartbeater.cc:511] Master 127.26.98.62:44455 requested a full tablet report, sending...
I20260619 02:49:36.926667 31643 ts_manager.cc:194] Registered new tserver with Master: a397df1bc5174ee792a1b9505414365d (127.26.98.1:45941)
W20260619 02:49:37.021412 31834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.021735 31834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.021777 31834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.023535 31834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.023615 31834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.023661 31834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:37.025614 31834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:37.026005 31834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.026391 31834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.032500 31840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.032523 31834 server_base.cc:1034] running on GCE node
W20260619 02:49:37.032500 31842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.032544 31839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.033219 31834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.033571 31834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.034785 31834 hybrid_clock.cc:648] HybridClock initialized: now 1781837377034755 us; error 48 us; skew 500 ppm
I20260619 02:49:37.036635 31834 webserver.cc:460] Webserver started at http://127.26.98.2:43025/ using document root <none> and password file <none>
I20260619 02:49:37.036929 31834 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.036981 31834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.037101 31834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.038594 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "806cf438364644c8ac3a99f1b6f013f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7a34b086a1a497ca4c801d901f1544a5"
server_key_iv: "5e82619225bf6ee15ad39c372c656796"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.039276 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "806cf438364644c8ac3a99f1b6f013f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7a34b086a1a497ca4c801d901f1544a5"
server_key_iv: "5e82619225bf6ee15ad39c372c656796"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.041054 31834 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:37.042162 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.042490 31834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:37.042600 31834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "806cf438364644c8ac3a99f1b6f013f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7a34b086a1a497ca4c801d901f1544a5"
server_key_iv: "5e82619225bf6ee15ad39c372c656796"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.042727 31834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.047205 31834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.047570 31834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.047720 31834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.047964 31834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:37.048367 31834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:37.048403 31834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.048439 31834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:37.048461 31834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.055985 31834 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:46601
I20260619 02:49:37.056051 31961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:46601 every 8 connection(s)
I20260619 02:49:37.056465 31834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:37.060418 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.057463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:55041 (local address 127.26.98.62:44455)
0619 02:49:37.057616 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.057619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.057840 (+ 221us) server_negotiation.cc:407] Connection header received
0619 02:49:37.058423 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.058425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.058431 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.058505 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.060128 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.060129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.060130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.060136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.060182 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.060187 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.060249 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.060331 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.060346 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 02:49:37.060676 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.057414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44455 (local address 127.26.98.2:55041)
0619 02:49:37.057647 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.057699 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.058302 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.058591 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.058595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.058615 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.059798 (+ 1183us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.059799 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.059849 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.059896 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.060256 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.060263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.060267 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.060340 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.060413 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260619 02:49:37.060772 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31834
I20260619 02:49:37.060879 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:37.061089 27016 external_mini_cluster.cc:1428] Setting key 501e9aac8b8ebde066aa37ba353f6e8f
I20260619 02:49:37.061204 31962 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44455
I20260619 02:49:37.061321 31962 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:37.061493 31962 heartbeater.cc:511] Master 127.26.98.62:44455 requested a full tablet report, sending...
I20260619 02:49:37.061880 31643 ts_manager.cc:194] Registered new tserver with Master: 806cf438364644c8ac3a99f1b6f013f0 (127.26.98.2:46601)
I20260619 02:49:37.062203 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--builtin_ntp_servers=127.26.98.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:37.163143 31965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.163478 31965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.163524 31965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.165197 31965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.165293 31965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.165343 31965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:37.167361 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:37.167758 31965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.168167 31965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:49:37.174660 31965 server_base.cc:1034] running on GCE node
W20260619 02:49:37.174613 31971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.174613 31973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.174605 31970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.175590 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.175972 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.177237 31965 hybrid_clock.cc:648] HybridClock initialized: now 1781837377177195 us; error 69 us; skew 500 ppm
I20260619 02:49:37.179195 31965 webserver.cc:460] Webserver started at http://127.26.98.3:35713/ using document root <none> and password file <none>
I20260619 02:49:37.179523 31965 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.179585 31965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.179747 31965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.181128 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "d7852abf222b42dfa85e8ed93cfc0fe6"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "58022a9c1fe8028aab3da072744eee03"
server_key_iv: "353b4696200742fe3310da41fa5deb10"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.181555 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "d7852abf222b42dfa85e8ed93cfc0fe6"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "58022a9c1fe8028aab3da072744eee03"
server_key_iv: "353b4696200742fe3310da41fa5deb10"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.183120 31965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 02:49:37.184034 31979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.184202 31965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:37.184322 31965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "d7852abf222b42dfa85e8ed93cfc0fe6"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "58022a9c1fe8028aab3da072744eee03"
server_key_iv: "353b4696200742fe3310da41fa5deb10"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.184396 31965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.188256 31965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.188598 31965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.188740 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.188959 31965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:37.189306 31965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:37.189340 31965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.189373 31965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:37.189395 31965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.199065 31965 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:37887
I20260619 02:49:37.199155 32092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:37887 every 8 connection(s)
I20260619 02:49:37.199656 31965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:37.203437 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.200700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:52007 (local address 127.26.98.62:44455)
0619 02:49:37.200761 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.200764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.201008 (+ 244us) server_negotiation.cc:407] Connection header received
0619 02:49:37.201619 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.201621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.201630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.201693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.203143 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.203144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.203145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.203152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.203173 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.203178 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.203233 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.203318 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.203338 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:37.203835 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.200639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44455 (local address 127.26.98.3:52007)
0619 02:49:37.200868 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.200911 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.201497 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.201787 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.201792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.201802 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.202931 (+ 1129us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.202931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.202982 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.203026 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.203238 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.203245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.203250 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.203331 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.203423 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260619 02:49:37.204599 32093 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44455
I20260619 02:49:37.204871 32093 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:37.205344 32093 heartbeater.cc:511] Master 127.26.98.62:44455 requested a full tablet report, sending...
I20260619 02:49:37.205965 31643 ts_manager.cc:194] Registered new tserver with Master: d7852abf222b42dfa85e8ed93cfc0fe6 (127.26.98.3:37887)
I20260619 02:49:37.209005 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 31965
I20260619 02:49:37.209151 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:37.209329 27016 external_mini_cluster.cc:1428] Setting key 722800b635c228a081178a585e64c429
I20260619 02:49:37.211033 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:37.214229 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.212782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.26.98.62:44455)
0619 02:49:37.212862 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.212864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.213066 (+ 202us) server_negotiation.cc:407] Connection header received
0619 02:49:37.213178 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.213179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.213187 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.213244 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.213963 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.213964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.213965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.213970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.213989 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.213993 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.214048 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.214099 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.214124 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:37.216807 31643 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:37.218814 31643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:37.224098 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.222071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46601 (local address 127.0.0.1:41148)
0619 02:49:37.222605 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.222614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.222651 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.223042 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.223043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.223045 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.223590 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.223590 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.223599 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.223613 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.223884 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.223893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.223897 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.223976 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.223992 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":2,"thread_start_us":147,"threads_started":1}
I20260619 02:49:37.224118 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.222399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.26.98.2:46601)
0619 02:49:37.222851 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.222855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.222865 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:37.222920 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.222922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.222932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.223191 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.223698 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.223705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.223705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.223709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.223752 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.223780 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.223883 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.223991 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.224009 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":98,"threads_started":1}
I20260619 02:49:37.224119 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.222087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.26.98.3:37887)
0619 02:49:37.222499 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.222505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.222517 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:37.222548 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.222550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.222562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.223103 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.223791 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.223795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.223795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.223809 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.223837 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.223854 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.223934 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.224013 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.224029 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":101,"threads_started":1}
I20260619 02:49:37.224102 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.221970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37887 (local address 127.0.0.1:53906)
0619 02:49:37.222206 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.222294 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.222364 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.222772 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.222780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.222790 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.223493 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.223494 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.223517 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.223549 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.223934 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.223935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.223936 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.223998 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.224013 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260619 02:49:37.225029 31896 tablet_service.cc:1467] Processing CreateTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 (DEFAULT_TABLE table=test-table [id=6ebe52b8ba2748bfbaafa445dd85c858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:37.225032 32027 tablet_service.cc:1467] Processing CreateTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 (DEFAULT_TABLE table=test-table [id=6ebe52b8ba2748bfbaafa445dd85c858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:37.225242 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.222071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:45941 (local address 127.0.0.1:40918)
0619 02:49:37.222476 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.222487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.222534 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.223022 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.223023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.223026 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.224748 (+ 1722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.224748 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.224758 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.224773 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.225081 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.225082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.225083 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.225153 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.225168 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":156,"threads_started":1}
I20260619 02:49:37.225255 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.222604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.26.98.1:45941)
0619 02:49:37.222850 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.222855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.222863 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:37.222923 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.222926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.222935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.223012 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.224875 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.224889 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.224889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.224895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.224943 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.224989 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.225075 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.225145 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.225164 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":117,"threads_started":1}
I20260619 02:49:37.225518 32027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b8861b478447c9a8084b9c16e0ace9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.225539 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b8861b478447c9a8084b9c16e0ace9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.227984 31765 tablet_service.cc:1467] Processing CreateTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 (DEFAULT_TABLE table=test-table [id=6ebe52b8ba2748bfbaafa445dd85c858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:37.228474 31765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b8861b478447c9a8084b9c16e0ace9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.229249 32107 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Bootstrap starting.
I20260619 02:49:37.230396 32107 tablet_bootstrap.cc:654] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.230808 32107 log.cc:824] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.231220 32108 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Bootstrap starting.
I20260619 02:49:37.231909 32107 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: No bootstrap required, opened a new log
I20260619 02:49:37.231999 32107 ts_tablet_manager.cc:1397] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:37.232033 32108 tablet_bootstrap.cc:654] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.232421 32108 log.cc:824] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.232609 32110 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Bootstrap starting.
I20260619 02:49:37.233436 32110 tablet_bootstrap.cc:654] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.233759 32110 log.cc:824] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.233871 32108 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: No bootstrap required, opened a new log
I20260619 02:49:37.233950 32108 ts_tablet_manager.cc:1397] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:37.234649 32110 tablet_bootstrap.cc:492] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: No bootstrap required, opened a new log
I20260619 02:49:37.234716 32110 ts_tablet_manager.cc:1397] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:37.234959 32107 raft_consensus.cc:348] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.235189 32107 raft_consensus.cc:374] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.235235 32107 raft_consensus.cc:729] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806cf438364644c8ac3a99f1b6f013f0, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.235405 32107 consensus_queue.cc:260] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.235754 32107 ts_tablet_manager.cc:1428] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:37.235952 31962 heartbeater.cc:503] Master 127.26.98.62:44455 was elected leader, sending a full tablet report...
I20260619 02:49:37.236829 32108 raft_consensus.cc:348] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.237044 32108 raft_consensus.cc:374] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.237071 32108 raft_consensus.cc:729] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7852abf222b42dfa85e8ed93cfc0fe6, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.237288 32108 consensus_queue.cc:260] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.237622 32108 ts_tablet_manager.cc:1428] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:37.237676 32093 heartbeater.cc:503] Master 127.26.98.62:44455 was elected leader, sending a full tablet report...
I20260619 02:49:37.237645 32110 raft_consensus.cc:348] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.237834 32110 raft_consensus.cc:374] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.237869 32110 raft_consensus.cc:729] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a397df1bc5174ee792a1b9505414365d, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.238003 32110 consensus_queue.cc:260] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.238365 31831 heartbeater.cc:503] Master 127.26.98.62:44455 was elected leader, sending a full tablet report...
I20260619 02:49:37.238366 32110 ts_tablet_manager.cc:1428] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 02:49:37.307286 31963 tablet.cc:2369] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:37.416276 32114 raft_consensus.cc:482] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:37.416462 32114 raft_consensus.cc:504] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.416978 32114 leader_election.cc:288] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 806cf438364644c8ac3a99f1b6f013f0 (127.26.98.2:46601), a397df1bc5174ee792a1b9505414365d (127.26.98.1:45941)
I20260619 02:49:37.418722 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.417466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:52609 (local address 127.26.98.1:45941)
0619 02:49:37.417571 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.417574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.417586 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:37.417657 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.417658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.417666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.417725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.418419 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.418422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.418422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.418428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.418452 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.418458 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.418529 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.418616 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.418632 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:49:37.418715 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.417302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:45941 (local address 127.26.98.3:52609)
0619 02:49:37.417407 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.417430 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.417555 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.417737 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.417738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.417741 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.418273 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.418274 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.418285 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.418312 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.418545 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.418546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.418546 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.418610 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.418628 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:49:37.418766 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.417414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:52603 (local address 127.26.98.2:46601)
0619 02:49:37.417558 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.417561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.417828 (+ 267us) server_negotiation.cc:407] Connection header received
0619 02:49:37.417884 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.417885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.417892 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.417945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.418509 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.418511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.418512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.418518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.418541 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.418547 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.418612 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.418685 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.418699 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:49:37.418851 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.417302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46601 (local address 127.26.98.3:52603)
0619 02:49:37.417737 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.417748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.417821 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.417955 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.417957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.417973 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.418404 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:37.418405 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.418413 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:37.418426 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.418607 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.418608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.418610 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.418687 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.418698 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":233,"thread_start_us":132,"threads_started":1}
I20260619 02:49:37.419235 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b8861b478447c9a8084b9c16e0ace9" candidate_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806cf438364644c8ac3a99f1b6f013f0" is_pre_election: true
I20260619 02:49:37.419234 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b8861b478447c9a8084b9c16e0ace9" candidate_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a397df1bc5174ee792a1b9505414365d" is_pre_election: true
I20260619 02:49:37.419467 31785 raft_consensus.cc:2454] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7852abf222b42dfa85e8ed93cfc0fe6 in term 0.
I20260619 02:49:37.419467 31916 raft_consensus.cc:2454] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7852abf222b42dfa85e8ed93cfc0fe6 in term 0.
I20260619 02:49:37.419771 31983 leader_election.cc:302] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806cf438364644c8ac3a99f1b6f013f0, d7852abf222b42dfa85e8ed93cfc0fe6; no voters:
I20260619 02:49:37.420002 32114 raft_consensus.cc:2781] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:37.420049 32114 raft_consensus.cc:482] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:37.420066 32114 raft_consensus.cc:3037] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.420950 32114 raft_consensus.cc:504] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.421123 32114 leader_election.cc:288] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [CANDIDATE]: Term 1 election: Requested vote from peers 806cf438364644c8ac3a99f1b6f013f0 (127.26.98.2:46601), a397df1bc5174ee792a1b9505414365d (127.26.98.1:45941)
I20260619 02:49:37.421456 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b8861b478447c9a8084b9c16e0ace9" candidate_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806cf438364644c8ac3a99f1b6f013f0"
I20260619 02:49:37.421442 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b8861b478447c9a8084b9c16e0ace9" candidate_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a397df1bc5174ee792a1b9505414365d"
I20260619 02:49:37.421577 31785 raft_consensus.cc:3037] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.421591 31916 raft_consensus.cc:3037] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:49:37.421789 31832 tablet.cc:2369] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:37.422789 31785 raft_consensus.cc:2454] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7852abf222b42dfa85e8ed93cfc0fe6 in term 1.
I20260619 02:49:37.422789 31916 raft_consensus.cc:2454] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7852abf222b42dfa85e8ed93cfc0fe6 in term 1.
I20260619 02:49:37.423130 31983 leader_election.cc:302] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806cf438364644c8ac3a99f1b6f013f0, d7852abf222b42dfa85e8ed93cfc0fe6; no voters:
I20260619 02:49:37.423243 32114 raft_consensus.cc:2781] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:37.423454 32114 raft_consensus.cc:686] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 1 LEADER]: Becoming Leader. State: Replica: d7852abf222b42dfa85e8ed93cfc0fe6, State: Running, Role: LEADER
I20260619 02:49:37.423571 32114 consensus_queue.cc:237] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } }
I20260619 02:49:37.424789 31643 catalog_manager.cc:5495] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7852abf222b42dfa85e8ed93cfc0fe6 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7852abf222b42dfa85e8ed93cfc0fe6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 } health_report { overall_health: UNKNOWN } } }
W20260619 02:49:37.450526 32094 tablet.cc:2369] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:37.461280 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.459900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.26.98.3:37887)
0619 02:49:37.459961 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.459964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.459993 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:49:37.460055 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.460056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.460081 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.460148 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.460946 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.460948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.460949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.460955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.460992 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.460997 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.461045 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.461165 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.461182 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:37.464179 31916 raft_consensus.cc:1261] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 1 FOLLOWER]: Refusing update from remote peer d7852abf222b42dfa85e8ed93cfc0fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:37.464179 31785 raft_consensus.cc:1261] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 1 FOLLOWER]: Refusing update from remote peer d7852abf222b42dfa85e8ed93cfc0fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:37.464748 32117 consensus_queue.cc:1035] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a397df1bc5174ee792a1b9505414365d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:37.464936 32114 consensus_queue.cc:1035] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806cf438364644c8ac3a99f1b6f013f0" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:37.469002 32123 mvcc.cc:205] Tried to move back new op lower bound from 7298405898089246720 to 7298405897927880704. Current Snapshot: MvccSnapshot[applied={T|T < 7298405898089246720}]
I20260619 02:49:37.472453 31642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:37.472596 31642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:37.473694 31642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9554b839434cdd9256fd8a3cfe748c: Sending DeleteTablet for 3 replicas of tablet 27b8861b478447c9a8084b9c16e0ace9
I20260619 02:49:37.474153 32027 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:37 UTC) from {username='slave'} at 127.0.0.1:53906
I20260619 02:49:37.474179 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:37 UTC) from {username='slave'} at 127.0.0.1:41148
I20260619 02:49:37.474272 31765 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b8861b478447c9a8084b9c16e0ace9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:37 UTC) from {username='slave'} at 127.0.0.1:40918
I20260619 02:49:37.474539 32132 tablet_replica.cc:330] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d: stopping tablet replica
I20260619 02:49:37.474720 32132 raft_consensus.cc:2229] T 27b8861b478447c9a8084b9c16e0ace9 P a397df1bc5174ee792a1b9505414365d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:37.474738 32131 tablet_replica.cc:330] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: stopping tablet replica
I20260619 02:49:37.474836 32131 raft_consensus.cc:2229] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:37.474982 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31703
I20260619 02:49:37.475039 32131 raft_consensus.cc:2258] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:37.476243 32130 tablet_replica.cc:330] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0: stopping tablet replica
I20260619 02:49:37.476404 32130 raft_consensus.cc:2229] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:37.476540 32130 raft_consensus.cc:2258] T 27b8861b478447c9a8084b9c16e0ace9 P 806cf438364644c8ac3a99f1b6f013f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:37.476889 32131 ts_tablet_manager.cc:1905] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:37.478986 32131 ts_tablet_manager.cc:1918] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:37.479106 32131 log.cc:1196] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/27b8861b478447c9a8084b9c16e0ace9
I20260619 02:49:37.479424 32131 ts_tablet_manager.cc:1939] T 27b8861b478447c9a8084b9c16e0ace9 P d7852abf222b42dfa85e8ed93cfc0fe6: Deleting consensus metadata
I20260619 02:49:37.480064 31627 catalog_manager.cc:4841] TS d7852abf222b42dfa85e8ed93cfc0fe6 (127.26.98.3:37887): tablet 27b8861b478447c9a8084b9c16e0ace9 (table test-table [id=6ebe52b8ba2748bfbaafa445dd85c858]) successfully deleted
W20260619 02:49:37.480602 31629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:45941 (error 108)
I20260619 02:49:37.480870 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31834
I20260619 02:49:37.481294 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.481146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:45941 (local address 127.0.0.1:40930)
0619 02:49:37.481211 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.481253 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:49:37.481549 31629 catalog_manager.cc:4584] TS a397df1bc5174ee792a1b9505414365d (127.26.98.1:45941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27b8861b478447c9a8084b9c16e0ace9: Network error: Client connection negotiation failed: client connection to 127.26.98.1:45941: connect: Connection refused (error 111)
I20260619 02:49:37.487475 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31965
I20260619 02:49:37.487620 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.487445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46601 (local address 127.0.0.1:41156)
0619 02:49:37.487526 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.487569 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:49:37.493176 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 31612
2026-06-19T02:49:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (920 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:49:37.515770 27016 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:37.517079 27016 webserver.cc:460] Webserver started at https://127.0.0.1:39745/ using document root <none> and password file <none>
I20260619 02:49:37.517203 27016 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:37.517400 27016 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
2026-06-19T02:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:37Z Disabled control of system clock
I20260619 02:49:37.527789 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44685
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39745/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/testchainca.cert with env {}
W20260619 02:49:37.625334 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.625665 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.625708 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.627839 32139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:37.627947 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.627965 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.627985 32139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:37.628000 32139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:37.628016 32139 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:37.628031 32139 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39745/jwks.json
I20260619 02:49:37.630443 32139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39745/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:37.630872 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.631188 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.636875 32145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.636874 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.636874 32144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.637262 32139 server_base.cc:1034] running on GCE node
I20260619 02:49:37.637529 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.637864 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.639015 32139 hybrid_clock.cc:648] HybridClock initialized: now 1781837377638992 us; error 42 us; skew 500 ppm
I20260619 02:49:37.640829 32139 webserver.cc:460] Webserver started at http://127.26.98.62:45359/ using document root <none> and password file <none>
I20260619 02:49:37.641140 32139 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.641201 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.641353 32139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.642733 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "9e723e9a622342a382044ab6b23960f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7324a6c0477d7ebffc07a2e243ba853e"
server_key_iv: "fd8a3e32b7012f71b5352567e2252393"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.643195 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "9e723e9a622342a382044ab6b23960f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7324a6c0477d7ebffc07a2e243ba853e"
server_key_iv: "fd8a3e32b7012f71b5352567e2252393"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.644850 32139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:37.645921 32153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.646183 32139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:37.646296 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "9e723e9a622342a382044ab6b23960f0"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "7324a6c0477d7ebffc07a2e243ba853e"
server_key_iv: "fd8a3e32b7012f71b5352567e2252393"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.646385 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.662954 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.663388 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.663542 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.668646 32139 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:44685
I20260619 02:49:37.669214 32205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:44685 every 8 connection(s)
I20260619 02:49:37.669256 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:37.670635 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.674098 32206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0: Bootstrap starting.
I20260619 02:49:37.674934 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32139
I20260619 02:49:37.675043 32206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.675076 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:37.675297 27016 external_mini_cluster.cc:1428] Setting key 590e8cea6d575495d62d88c86990af14
I20260619 02:49:37.675424 32206 log.cc:824] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.676627 32206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0: No bootstrap required, opened a new log
I20260619 02:49:37.679752 32206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } }
I20260619 02:49:37.680071 32206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.680116 32206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e723e9a622342a382044ab6b23960f0, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.680322 32206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } }
I20260619 02:49:37.680419 32206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:37.680454 32206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:37.680501 32206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.681593 32206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } }
I20260619 02:49:37.681748 32206 leader_election.cc:302] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e723e9a622342a382044ab6b23960f0; no voters:
I20260619 02:49:37.681972 32206 leader_election.cc:288] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:37.682216 32211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:37.682369 32211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [term 1 LEADER]: Becoming Leader. State: Replica: 9e723e9a622342a382044ab6b23960f0, State: Running, Role: LEADER
I20260619 02:49:37.682372 32206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:37.682516 32211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } }
I20260619 02:49:37.682957 32212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e723e9a622342a382044ab6b23960f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } } }
I20260619 02:49:37.683063 32212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.683279 32213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e723e9a622342a382044ab6b23960f0. Latest consensus state: current_term: 1 leader_uuid: "9e723e9a622342a382044ab6b23960f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e723e9a622342a382044ab6b23960f0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44685 } } }
I20260619 02:49:37.683337 32213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.683590 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.676198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.26.98.62:44685)
0619 02:49:37.676457 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.676466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.676502 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:49:37.677155 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.677166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.677360 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.677697 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.677985 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.679249 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.680555 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.681282 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.681460 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.682422 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.682428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.682429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.682741 (+ 312us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.682804 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.682824 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.682932 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.683169 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.683303 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":110,"threads_started":1}
I20260619 02:49:37.683395 32218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:37.685086 32218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:37.685194 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:37.687217 32218 catalog_manager.cc:1347] Generated new cluster ID: 3a8376b67fcf40179f987cd1925aaf3e
I20260619 02:49:37.687405 32218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:37.711026 32218 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:37.712030 32218 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:37.724460 32218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e723e9a622342a382044ab6b23960f0: Generated new TSK 0
I20260619 02:49:37.724757 32218 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:49:37.755712 32236 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:49:37.755936 32236 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:49:37.756146 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.738776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.26.98.62:44685)
0619 02:49:37.738837 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.738840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.738995 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:49:37.739107 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.739108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.739124 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.739226 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:37.739408 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.740256 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.741219 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.741512 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.741592 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.742076 (+ 484us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:37.755862 (+ 13786us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:49:37.756057 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 02:49:37.756219 32210 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:49:37.756700 32235 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:37.738684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44685 (local address 127.0.0.1:58254)
0619 02:49:37.738910 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.738924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.739004 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.739252 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.739254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.739256 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:37.739338 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.739341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.740399 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.740400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.741127 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.741131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.741724 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.741726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.741827 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.741871 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:37.756197 (+ 14326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:49:37.756380 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.98.62:44685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.26.98.62:44685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:49:37.757342 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32139
2026-06-19T02:49:37Z chronyd exiting
I20260619 02:49:37.794665 27016 test_util.cc:175] -----------------------------------------------
I20260619 02:49:37.794798 27016 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837352638110-27016-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:49:37.795955 27016 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:37.796806 27016 webserver.cc:460] Webserver started at https://127.0.0.1:35847/ using document root <none> and password file <none>
I20260619 02:49:37.796861 27016 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:37.797003 27016 webserver.cc:460] Webserver started at http://127.0.0.1:43503/ using document root <none> and password file <none>
2026-06-19T02:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:37Z Disabled control of system clock
I20260619 02:49:37.807343 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43535
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:43535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35847/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/testchainca.cert with env {}
W20260619 02:49:37.908969 32242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.909317 32242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.909350 32242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.911041 32242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:37.911121 32242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.911134 32242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.911145 32242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:37.911159 32242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:37.911171 32242 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:37.911182 32242 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35847/jwks.json
I20260619 02:49:37.914438 32242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:43535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35847/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:37.914801 32242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.915063 32242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:49:37.920682 32242 server_base.cc:1034] running on GCE node
W20260619 02:49:37.920751 32247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.920785 32250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.920874 32248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.921307 32242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.921649 32242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.922847 32242 hybrid_clock.cc:648] HybridClock initialized: now 1781837377922822 us; error 39 us; skew 500 ppm
I20260619 02:49:37.924736 32242 webserver.cc:460] Webserver started at http://127.26.98.62:38867/ using document root <none> and password file <none>
I20260619 02:49:37.925035 32242 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.925237 32242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.925400 32242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.926775 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "e06e60aa182d49c88f94fe2cf5141888"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "1d0cdbc6fdb3b28ff8eca2436ca551c1"
server_key_iv: "3279b75ad360c1517710c671cec0a8bd"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.927352 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "e06e60aa182d49c88f94fe2cf5141888"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "1d0cdbc6fdb3b28ff8eca2436ca551c1"
server_key_iv: "3279b75ad360c1517710c671cec0a8bd"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.929565 32242 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:37.930631 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.930842 32242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:37.930981 32242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "e06e60aa182d49c88f94fe2cf5141888"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-0f9d"
server_key: "1d0cdbc6fdb3b28ff8eca2436ca551c1"
server_key_iv: "3279b75ad360c1517710c671cec0a8bd"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.931087 32242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.941565 32242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.941989 32242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.942137 32242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.947242 32242 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:43535
I20260619 02:49:37.947369 32308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:43535 every 8 connection(s)
I20260619 02:49:37.947758 32242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:37.948700 32309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.952473 32309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: Bootstrap starting.
I20260619 02:49:37.953279 32309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.953619 32309 log.cc:824] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.953900 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32242
I20260619 02:49:37.954123 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:37.954447 27016 external_mini_cluster.cc:1428] Setting key 3726f1ecd79998a5d2c68869468f7beb
I20260619 02:49:37.954944 32309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: No bootstrap required, opened a new log
I20260619 02:49:37.957103 32309 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } }
I20260619 02:49:37.957298 32309 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.957322 32309 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06e60aa182d49c88f94fe2cf5141888, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.957463 32309 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } }
I20260619 02:49:37.957561 32309 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:37.957600 32309 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:37.957645 32309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.958734 32309 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } }
I20260619 02:49:37.958915 32309 leader_election.cc:302] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06e60aa182d49c88f94fe2cf5141888; no voters:
I20260619 02:49:37.959132 32309 leader_election.cc:288] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:37.959183 32314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:37.959383 32314 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [term 1 LEADER]: Becoming Leader. State: Replica: e06e60aa182d49c88f94fe2cf5141888, State: Running, Role: LEADER
I20260619 02:49:37.959451 32309 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:37.959522 32314 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } }
I20260619 02:49:37.961133 32315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e06e60aa182d49c88f94fe2cf5141888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } } }
I20260619 02:49:37.961469 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06e60aa182d49c88f94fe2cf5141888. Latest consensus state: current_term: 1 leader_uuid: "e06e60aa182d49c88f94fe2cf5141888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06e60aa182d49c88f94fe2cf5141888" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43535 } } }
I20260619 02:49:37.961777 32315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.961786 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888 [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:37.961696 32329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:37.962280 32329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:37.962430 32331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:37.962666 32331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:37.963990 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.955186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.26.98.62:43535)
0619 02:49:37.955495 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.955506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.955547 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:37.957084 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.957098 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.957214 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.957432 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.957740 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.959664 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.960933 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.961643 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.962042 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.962943 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.962950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.962951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.963245 (+ 294us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.963319 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.963341 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.963441 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.963609 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.963725 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260619 02:49:37.965260 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:37.966034 32331 catalog_manager.cc:1347] Generated new cluster ID: cb00f3831792499daeb5263133ac5275
I20260619 02:49:37.966111 32331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:37.980880 32331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:37.981534 32331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:37.988265 32331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e06e60aa182d49c88f94fe2cf5141888: Generated new TSK 0
I20260619 02:49:37.988550 32331 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:49:38.016134 32339 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:49:38.016325 32339 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:49:38.016705 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.999858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.26.98.62:43535)
0619 02:49:37.999935 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.999938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.000001 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:49:38.000075 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.000076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.000092 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.000212 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:38.000498 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.001359 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.002379 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.002704 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.002786 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.003080 (+ 294us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:38.016400 (+ 13320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:49:38.016607 (+ 207us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 02:49:38.016781 32312 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 02:49:38.016795 32338 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:37.999752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43535 (local address 127.0.0.1:56768)
0619 02:49:37.999926 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.999936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:38.000004 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:38.000317 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:38.000318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:38.000321 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:38.000405 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.000408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.001498 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.001500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.002260 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.002264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.002817 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.002818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.002927 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.002937 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:38.016564 (+ 13627us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:49:38.016707 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.98.62:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260619 02:49:38.017182 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32242
2026-06-19T02:49:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (260 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:49:38.056412 27016 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:38.057227 27016 webserver.cc:460] Webserver started at https://127.0.0.1:44163/ using document root <none> and password file <none>
I20260619 02:49:38.057283 27016 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:38.057430 27016 webserver.cc:460] Webserver started at http://127.0.0.1:43247/ using document root <none> and password file <none>
2026-06-19T02:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:38Z Disabled control of system clock
I20260619 02:49:38.067312 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39345
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39345
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44163/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:49:38.180624 32345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:38.181010 32345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:38.181080 32345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:38.183622 32345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:38.183705 32345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:38.183724 32345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:38.183740 32345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:38.183756 32345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:38.183773 32345 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:38.183799 32345 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44163/jwks.json
I20260619 02:49:38.185632 32345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44163/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:38.185974 32345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:38.187250 32345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:38.194785 32353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:38.194785 32350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:38.195057 32351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:38.195407 32345 server_base.cc:1034] running on GCE node
I20260619 02:49:38.195729 32345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:38.196116 32345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:38.197351 32345 hybrid_clock.cc:648] HybridClock initialized: now 1781837378197319 us; error 48 us; skew 500 ppm
I20260619 02:49:38.199545 32345 webserver.cc:460] Webserver started at http://127.26.98.62:45723/ using document root <none> and password file <none>
I20260619 02:49:38.199824 32345 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:38.199952 32345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:38.200095 32345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:38.201274 32345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "6ff171315c4e43809076bde5c9b9c8bc"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "67253a651520592d0b085bfd0936f0a2"
server_key_iv: "ade1cd5a3896002858b1d8a3e969c426"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.201620 32345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "6ff171315c4e43809076bde5c9b9c8bc"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "67253a651520592d0b085bfd0936f0a2"
server_key_iv: "ade1cd5a3896002858b1d8a3e969c426"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.203356 32345 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 02:49:38.204587 32359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:38.204999 32345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:38.205139 32345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "6ff171315c4e43809076bde5c9b9c8bc"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "67253a651520592d0b085bfd0936f0a2"
server_key_iv: "ade1cd5a3896002858b1d8a3e969c426"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.205248 32345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:38.219853 32345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:38.220391 32345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:38.220582 32345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:38.226046 32345 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:39345
I20260619 02:49:38.226267 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:39345 every 8 connection(s)
I20260619 02:49:38.226568 32345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:38.227550 32412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:38.231052 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc: Bootstrap starting.
I20260619 02:49:38.231755 32412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:38.232025 32412 log.cc:824] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:38.232797 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc: No bootstrap required, opened a new log
I20260619 02:49:38.234802 32412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } }
I20260619 02:49:38.235107 32412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:38.235150 32412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff171315c4e43809076bde5c9b9c8bc, State: Initialized, Role: FOLLOWER
I20260619 02:49:38.235188 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32345
I20260619 02:49:38.235291 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:38.235390 32412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } }
I20260619 02:49:38.235482 27016 external_mini_cluster.cc:1428] Setting key 4d0f104f3f0a7307212271d7231cda88
I20260619 02:49:38.235491 32412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:38.235528 32412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:38.235574 32412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:38.236510 32412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } }
I20260619 02:49:38.236647 32412 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff171315c4e43809076bde5c9b9c8bc; no voters:
I20260619 02:49:38.236822 32412 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:38.236857 32417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:38.237012 32417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [term 1 LEADER]: Becoming Leader. State: Replica: 6ff171315c4e43809076bde5c9b9c8bc, State: Running, Role: LEADER
I20260619 02:49:38.237118 32417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } }
I20260619 02:49:38.237118 32412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:38.237825 32418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff171315c4e43809076bde5c9b9c8bc. Latest consensus state: current_term: 1 leader_uuid: "6ff171315c4e43809076bde5c9b9c8bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } } }
I20260619 02:49:38.237910 32418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.238374 32419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff171315c4e43809076bde5c9b9c8bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff171315c4e43809076bde5c9b9c8bc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39345 } } }
I20260619 02:49:38.238448 32419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.238533 32426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:38.239158 32426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:38.241951 32426 catalog_manager.cc:1347] Generated new cluster ID: b4c90f7179384588b7aa72d79c66f421
I20260619 02:49:38.242033 32426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:38.242908 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:38.236124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.26.98.62:39345)
0619 02:49:38.236408 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:38.236418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.236459 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:38.236610 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.236618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.236719 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.236908 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:38.237426 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.239424 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.240400 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.241024 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.241184 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.241972 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:38.241977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:38.241977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:38.242229 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:38.242280 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:38.242336 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:38.242404 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:38.242523 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:49:38.242625 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":122,"threads_started":1}
I20260619 02:49:38.244319 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:38.251807 32426 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:38.253279 32426 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:38.258538 32426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff171315c4e43809076bde5c9b9c8bc: Generated new TSK 0
I20260619 02:49:38.258790 32426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:38.274662 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:38.270984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.26.98.62:39345)
0619 02:49:38.271177 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:38.271179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.271189 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:38.271245 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.271246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.271259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.271332 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:38.271475 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.272063 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.272912 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.273143 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.273205 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.273414 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:38.274418 (+ 1004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:49:38.274569 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60368: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":158}
W20260619 02:49:38.274729 32441 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:38.270802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39345 (local address 127.0.0.1:60368)
0619 02:49:38.271031 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:38.271044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:38.271118 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:38.271333 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:38.271334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:38.271336 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:38.271410 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.271413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.272155 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.272156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.272805 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.272806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.273223 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.273224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.273287 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.273295 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:38.274518 (+ 1223us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:49:38.274627 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.98.62:39345: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":154,"threads_started":1}
W20260619 02:49:38.274767 32416 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60368: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 02:49:38.275049 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32345
E20260619 02:49:38.288091 32442 webserver.cc:533] Webserver: sslize failed:
E20260619 02:49:38.288219 32442 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T02:49:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:38Z Disabled control of system clock
I20260619 02:49:38.325049 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34693
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:34693
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/testchainca.cert with env {}
W20260619 02:49:38.428282 32446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:38.428620 32446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:38.428649 32446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:38.430662 32446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:38.430778 32446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:38.430792 32446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:38.430806 32446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:38.430817 32446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:38.430831 32446 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:38.430841 32446 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:49:38.433276 32446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:34693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:38.433648 32446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:38.433905 32446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:38.439759 32451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:38.439760 32454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:38.439759 32452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:38.440191 32446 server_base.cc:1034] running on GCE node
I20260619 02:49:38.440528 32446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:38.440934 32446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:38.442130 32446 hybrid_clock.cc:648] HybridClock initialized: now 1781837378442112 us; error 44 us; skew 500 ppm
I20260619 02:49:38.443921 32446 webserver.cc:460] Webserver started at http://127.26.98.62:38351/ using document root <none> and password file <none>
I20260619 02:49:38.444268 32446 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:38.444331 32446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:38.444478 32446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:38.445814 32446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "24103e13f9514bd4b9ecff681ab93b22"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "0a2dcfd6aa544da4a7d11f48b56ca70b"
server_key_iv: "848a629cf28459f70156d9db020b3fc5"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.446244 32446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "24103e13f9514bd4b9ecff681ab93b22"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "0a2dcfd6aa544da4a7d11f48b56ca70b"
server_key_iv: "848a629cf28459f70156d9db020b3fc5"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.447924 32446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260619 02:49:38.448962 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:38.449276 32446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:38.449383 32446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "24103e13f9514bd4b9ecff681ab93b22"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-0f9d"
server_key: "0a2dcfd6aa544da4a7d11f48b56ca70b"
server_key_iv: "848a629cf28459f70156d9db020b3fc5"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.449467 32446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:38.458863 32446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:38.459488 32446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:38.459654 32446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:38.464557 32446 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:34693
I20260619 02:49:38.464684 32512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:34693 every 8 connection(s)
I20260619 02:49:38.465052 32446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:38.466229 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:38.469628 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22: Bootstrap starting.
I20260619 02:49:38.470403 32513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:38.470728 32513 log.cc:824] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:38.471833 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22: No bootstrap required, opened a new log
I20260619 02:49:38.472038 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32446
I20260619 02:49:38.472153 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:38.472360 27016 external_mini_cluster.cc:1428] Setting key 2007e5fc807e678e8dfb35629f468d21
I20260619 02:49:38.474749 32513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } }
I20260619 02:49:38.475121 32513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:38.475168 32513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24103e13f9514bd4b9ecff681ab93b22, State: Initialized, Role: FOLLOWER
I20260619 02:49:38.475328 32513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } }
I20260619 02:49:38.475414 32513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:38.475447 32513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:38.475497 32513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:38.476249 32513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } }
I20260619 02:49:38.476366 32513 leader_election.cc:302] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24103e13f9514bd4b9ecff681ab93b22; no voters:
I20260619 02:49:38.476568 32513 leader_election.cc:288] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:38.476613 32518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:38.476737 32518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [term 1 LEADER]: Becoming Leader. State: Replica: 24103e13f9514bd4b9ecff681ab93b22, State: Running, Role: LEADER
I20260619 02:49:38.476835 32518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } }
I20260619 02:49:38.476928 32513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:38.477326 32520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24103e13f9514bd4b9ecff681ab93b22. Latest consensus state: current_term: 1 leader_uuid: "24103e13f9514bd4b9ecff681ab93b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } } }
I20260619 02:49:38.477595 32520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.477530 32519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24103e13f9514bd4b9ecff681ab93b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24103e13f9514bd4b9ecff681ab93b22" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34693 } } }
I20260619 02:49:38.477895 32519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.478107 32528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:38.479323 32528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:38.481326 32528 catalog_manager.cc:1347] Generated new cluster ID: ab697c47fbbf438c9c112d1e5bfd872f
I20260619 02:49:38.481402 32528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:38.481812 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:38.473092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.26.98.62:34693)
0619 02:49:38.473439 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:38.473452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.473495 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:49:38.474680 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.474694 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.474811 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.475048 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:38.475499 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.477607 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.479076 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.479753 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.479904 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.480815 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:38.480821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:38.480822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:38.481075 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:38.481125 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:38.481143 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:38.481215 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:38.481451 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 02:49:38.481558 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":163,"threads_started":1}
I20260619 02:49:38.482963 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:38.491403 32528 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:38.492036 32528 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:38.496625 32528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24103e13f9514bd4b9ecff681ab93b22: Generated new TSK 0
I20260619 02:49:38.496907 32528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:38.965196 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:38.507911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.26.98.62:34693)
0619 02:49:38.507980 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:38.507982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.508038 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:49:38.508155 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.508156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.508187 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.508282 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:38.508497 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.509326 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.510607 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.510964 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.511043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.511301 (+ 258us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:38.964864 (+453563us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:49:38.965116 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 02:49:38.965287 32517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 02:49:38.965546 32542 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:38.507790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34693 (local address 127.0.0.1:45614)
0619 02:49:38.507968 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:38.507980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:38.508049 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:38.508323 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:38.508324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:38.508327 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:38.508399 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.508407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.509456 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.509458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.510445 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:38.510452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.511100 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.511101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:38.511162 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.511170 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:38.965179 (+454009us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:49:38.965405 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.98.62:34693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260619 02:49:38.965986 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32446
2026-06-19T02:49:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (669 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (106 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:39 dist-test-slave-0f9d krb5kdc[32554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:39 dist-test-slave-0f9d krb5kdc[32554](info): set up 2 sockets
Jun 19 02:49:39 dist-test-slave-0f9d krb5kdc[32554](info): commencing operation
krb5kdc: starting...
W20260619 02:49:40.179745 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:40.316346 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:38595
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:40.412919 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:40.413247 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:40.413288 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:40.414868 32567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:40.414980 32567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:40.415000 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:40.415019 32567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:40.415035 32567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:40.417763 32567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:40.418290 32567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:40.418596 32567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:40.424559 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.424566 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.424585 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:40.425056 32567 server_base.cc:1034] running on GCE node
I20260619 02:49:40.425441 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:40.425836 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:40.426997 32567 hybrid_clock.cc:648] HybridClock initialized: now 1781837380426971 us; error 46 us; skew 500 ppm
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:40.430941 32567 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:40.431689 32567 webserver.cc:460] Webserver started at http://127.26.98.62:45171/ using document root <none> and password file <none>
I20260619 02:49:40.431999 32567 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:40.432065 32567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:40.432236 32567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:40.433914 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "45f30bacc82b43debdb5e083df75e204"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "b7ca8ed54902f3463eb58b9bedec2f7d"
server_key_iv: "bf797b6f9335273659d9e17d25379150"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.434430 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "45f30bacc82b43debdb5e083df75e204"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "b7ca8ed54902f3463eb58b9bedec2f7d"
server_key_iv: "bf797b6f9335273659d9e17d25379150"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.436643 32567 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260619 02:49:40.437825 32582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.438122 32567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:40.438246 32567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "45f30bacc82b43debdb5e083df75e204"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "b7ca8ed54902f3463eb58b9bedec2f7d"
server_key_iv: "bf797b6f9335273659d9e17d25379150"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.438341 32567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:40.457583 32567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:40.459893 32567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:40.460289 32567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:40.465559 32567 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:38595
I20260619 02:49:40.465735 32634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:38595 every 8 connection(s)
I20260619 02:49:40.466061 32567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:40.467501 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:40.470849 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204: Bootstrap starting.
I20260619 02:49:40.471698 32635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:40.472033 32635 log.cc:824] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:40.473166 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204: No bootstrap required, opened a new log
I20260619 02:49:40.473616 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32567
I20260619 02:49:40.473730 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:40.473956 27016 external_mini_cluster.cc:1428] Setting key 9de0a4ff6328d96c149fa1b1c7c60557
I20260619 02:49:40.475996 32635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } }
I20260619 02:49:40.476258 32635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:40.476296 32635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f30bacc82b43debdb5e083df75e204, State: Initialized, Role: FOLLOWER
I20260619 02:49:40.476485 32635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } }
I20260619 02:49:40.476567 32635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:40.476620 32635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:40.476689 32635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:40.478065 32635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } }
I20260619 02:49:40.478240 32635 leader_election.cc:302] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f30bacc82b43debdb5e083df75e204; no voters:
I20260619 02:49:40.478768 32635 leader_election.cc:288] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:40.478823 32640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:40.479039 32640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [term 1 LEADER]: Becoming Leader. State: Replica: 45f30bacc82b43debdb5e083df75e204, State: Running, Role: LEADER
I20260619 02:49:40.479260 32640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } }
I20260619 02:49:40.479404 32635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:40.479796 32640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f30bacc82b43debdb5e083df75e204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } } }
I20260619 02:49:40.479962 32640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:40.480033 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f30bacc82b43debdb5e083df75e204. Latest consensus state: current_term: 1 leader_uuid: "45f30bacc82b43debdb5e083df75e204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f30bacc82b43debdb5e083df75e204" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 38595 } } }
I20260619 02:49:40.480233 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:40.480360 32647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:40.481127 32647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:40.483255 32647 catalog_manager.cc:1347] Generated new cluster ID: b52b0912ceba49a792e601a7a609b98e
I20260619 02:49:40.483347 32647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:40.492848 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.474914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.26.98.62:38595)
0619 02:49:40.475242 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:40.475254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:40.475296 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:49:40.475962 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:40.475975 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:40.476069 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:40.476325 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:40.477219 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.478406 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.479968 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.480718 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.480872 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:40.488115 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:40.488124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:40.488124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:40.488154 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:40.490840 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.491368 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.491373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.491373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.491497 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.491694 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.491695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.491695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.492025 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:40.492218 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:40.492347 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:49:40.492535 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":144,"threads_started":1}
I20260619 02:49:40.493808 32647 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:40.495108 32647 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:40.499290 32647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f30bacc82b43debdb5e083df75e204: Generated new TSK 0
I20260619 02:49:40.499559 32647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:40.541939 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:40.665144 32661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:40.665457 32661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:40.665486 32661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:40.666828 32661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:40.666903 32661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:40.666942 32661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:40.668812 32661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:40.669138 32661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:40.669421 32661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:40.675902 32667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.675887 32666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.675930 32669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:40.676184 32661 server_base.cc:1034] running on GCE node
I20260619 02:49:40.676594 32661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:40.677000 32661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:40.678257 32661 hybrid_clock.cc:648] HybridClock initialized: now 1781837380678235 us; error 39 us; skew 500 ppm
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:40.681890 32661 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:40.682475 32661 webserver.cc:460] Webserver started at http://127.26.98.1:34227/ using document root <none> and password file <none>
I20260619 02:49:40.682734 32661 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:40.682792 32661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:40.682986 32661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:40.684362 32661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "e761ebb875714d3faa2e8b880a783f43"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0804e9c41d2a90c9f5e86605352a992c"
server_key_iv: "d14187a298bf894d5f5339be76a03396"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.684785 32661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "e761ebb875714d3faa2e8b880a783f43"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0804e9c41d2a90c9f5e86605352a992c"
server_key_iv: "d14187a298bf894d5f5339be76a03396"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.686604 32661 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:40.687722 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.687924 32661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:40.688025 32661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "e761ebb875714d3faa2e8b880a783f43"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0804e9c41d2a90c9f5e86605352a992c"
server_key_iv: "d14187a298bf894d5f5339be76a03396"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.688117 32661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:40.716091 32661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:40.719321 32661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:40.719496 32661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:40.719780 32661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:40.720148 32661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:40.720184 32661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.720220 32661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:40.720245 32661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.728081 32661 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:34001
I20260619 02:49:40.728139 321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:34001 every 8 connection(s)
I20260619 02:49:40.728564 32661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:40.730563 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 32661
I20260619 02:49:40.730690 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:40.730939 27016 external_mini_cluster.cc:1428] Setting key 222ec3ee3700bae3dfc24c2f1f00b306
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:49:40.744637 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.729875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35441 (local address 127.26.98.62:38595)
0619 02:49:40.730239 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:40.730241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:40.730259 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:49:40.731558 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:40.731560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:40.731574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:40.731627 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:40.732781 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.733660 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.735892 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.736354 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.736442 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:40.740561 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:40.740564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:40.740565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:40.740584 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:40.742482 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.743127 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.743129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.743129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.743227 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.743635 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.743638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.743638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.743841 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:40.744043 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:40.744406 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 02:49:40.744517 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260619 02:49:40.744793 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.729607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:38595 (local address 127.26.98.1:35441)
0619 02:49:40.729843 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:40.729917 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:40.731401 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:40.731766 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:40.731772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:40.732259 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:40.732643 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:40.732653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.733862 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.733865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:40.735727 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:40.735736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.736472 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.736473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:40.736915 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:40.737655 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:40.737666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:40.740408 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:40.742617 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:40.742629 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:40.742632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:40.743000 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:40.743300 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:40.743300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:40.743301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:40.743432 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:40.744050 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:40.744057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:40.744251 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 02:49:40.744362 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:49:40.744539 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
I20260619 02:49:40.745631 322 heartbeater.cc:347] Connected to a master server at 127.26.98.62:38595
I20260619 02:49:40.745824 322 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:40.746115 322 heartbeater.cc:511] Master 127.26.98.62:38595 requested a full tablet report, sending...
I20260619 02:49:40.746980 32599 ts_manager.cc:194] Registered new tserver with Master: e761ebb875714d3faa2e8b880a783f43 (127.26.98.1:34001)
I20260619 02:49:40.747897 32599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:35441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:40.767169 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:40.871608 327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:40.871906 327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:40.871954 327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:40.873466 327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:40.873533 327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:40.873579 327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:40.875694 327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:40.876084 327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:40.876381 327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:40.882812 333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.882718 332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.883176 335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:40.883383 327 server_base.cc:1034] running on GCE node
I20260619 02:49:40.883695 327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:40.884032 327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:40.885353 327 hybrid_clock.cc:648] HybridClock initialized: now 1781837380885200 us; error 62 us; skew 500 ppm
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:40.889041 327 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:40.889947 327 webserver.cc:460] Webserver started at http://127.26.98.2:34537/ using document root <none> and password file <none>
I20260619 02:49:40.890364 327 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:40.890467 327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:40.890825 327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:40.892305 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "ffb705557a1a461f98a7f33ae3a63ae8"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0cb79fb796f6e96dfd500d58e8e03d19"
server_key_iv: "390eca9bfe28e908afb21ec7e0698e97"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.892720 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb705557a1a461f98a7f33ae3a63ae8"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0cb79fb796f6e96dfd500d58e8e03d19"
server_key_iv: "390eca9bfe28e908afb21ec7e0698e97"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.894634 327 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:40.895746 342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.895946 327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:40.896049 327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "ffb705557a1a461f98a7f33ae3a63ae8"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-0f9d"
server_key: "0cb79fb796f6e96dfd500d58e8e03d19"
server_key_iv: "390eca9bfe28e908afb21ec7e0698e97"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.896134 327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:40.934073 327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:40.936981 327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:40.937170 327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:40.937481 327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:40.937848 327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:40.937887 327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.937922 327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:40.937939 327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.945708 327 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:45253
I20260619 02:49:40.945813 455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:45253 every 8 connection(s)
I20260619 02:49:40.946215 327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:40 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:40.955269 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 327
I20260619 02:49:40.955390 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:40.955606 27016 external_mini_cluster.cc:1428] Setting key 269db59dbcdcc347d77a2772c2ca1733
I20260619 02:49:40.959084 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.947319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:35835 (local address 127.26.98.62:38595)
0619 02:49:40.947414 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:40.947417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:40.947594 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:49:40.948720 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:40.948722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:40.948737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:40.948803 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:40.949612 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.950187 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.951558 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.951801 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.951941 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:40.954798 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:40.954801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:40.954801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:40.954812 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:40.956581 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.957525 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.957527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.957528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.957619 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:40.958045 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:40.958047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:40.958048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:40.958296 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:40.958404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:40.958828 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:49:40.958996 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:49:40.959338 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.947237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:38595 (local address 127.26.98.2:35835)
0619 02:49:40.947430 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:40.947479 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:40.948562 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:40.948899 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:40.948904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:40.949247 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:40.949499 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:40.949505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.950313 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.950314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:40.951381 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:40.951385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.951998 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:40.952000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:40.952160 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:40.952811 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:40.952822 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:40.954637 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:40.956711 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:40.956730 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:40.956734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:40.957279 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:40.957696 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:40.957697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:40.957697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:40.957865 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:40.958405 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:40.958413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:40.958644 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 02:49:40.958785 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 02:49:40.959064 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260619 02:49:40.959951 456 heartbeater.cc:347] Connected to a master server at 127.26.98.62:38595
I20260619 02:49:40.960103 456 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:40.960361 456 heartbeater.cc:511] Master 127.26.98.62:38595 requested a full tablet report, sending...
I20260619 02:49:40.960840 32599 ts_manager.cc:194] Registered new tserver with Master: ffb705557a1a461f98a7f33ae3a63ae8 (127.26.98.2:45253)
I20260619 02:49:40.961298 32599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:35835
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:40.986938 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:41.089360 461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:41.089702 461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:41.089733 461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:41.091476 461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:41.091573 461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:41.091626 461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:41.094334 461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:41.094866 461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:41.095305 461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:41.101953 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.101940 466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.101948 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.102365 461 server_base.cc:1034] running on GCE node
I20260619 02:49:41.102622 461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:41.102958 461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:41.104295 461 hybrid_clock.cc:648] HybridClock initialized: now 1781837381104267 us; error 35 us; skew 500 ppm
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.108009 461 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:41.108649 461 webserver.cc:460] Webserver started at http://127.26.98.3:37679/ using document root <none> and password file <none>
I20260619 02:49:41.108908 461 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:41.108968 461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:41.109122 461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:41.110462 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "8379e12719bc40969d759da2f7ab5df0"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-0f9d"
server_key: "82cd2d97ea6faa859a9d72af569c8fb6"
server_key_iv: "98653dddfba1398fbf437ba15c8d8694"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.111047 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "8379e12719bc40969d759da2f7ab5df0"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-0f9d"
server_key: "82cd2d97ea6faa859a9d72af569c8fb6"
server_key_iv: "98653dddfba1398fbf437ba15c8d8694"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.112886 461 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:41.114048 476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.114346 461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:41.114461 461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "8379e12719bc40969d759da2f7ab5df0"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-0f9d"
server_key: "82cd2d97ea6faa859a9d72af569c8fb6"
server_key_iv: "98653dddfba1398fbf437ba15c8d8694"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.114562 461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:41.130218 461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:41.133062 461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:41.133266 461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:41.133566 461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:41.133962 461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:41.134001 461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.134037 461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:41.134055 461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.142776 461 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:40843
I20260619 02:49:41.142815 589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:40843 every 8 connection(s)
I20260619 02:49:41.143361 461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:41.143860 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 461
I20260619 02:49:41.143945 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:41.144160 27016 external_mini_cluster.cc:1428] Setting key a8e707bdc04580afb0b758857cb6a59c
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:41.158187 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.144578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:47713 (local address 127.26.98.62:38595)
0619 02:49:41.144639 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.144641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.144970 (+ 329us) server_negotiation.cc:407] Connection header received
0619 02:49:41.146441 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.146442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.146461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.146526 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.148001 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.148576 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.150432 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.150661 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.150831 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.154446 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.154449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.154449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.154460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.156425 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.157069 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.157071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.157072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.157140 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.157402 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.157404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.157404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.157542 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.157605 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.157960 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.158077 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:49:41.158437 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.144554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:38595 (local address 127.26.98.3:47713)
0619 02:49:41.144766 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.144828 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.146212 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.146757 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.146764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.147504 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.147809 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.147818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.148871 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.148873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.150143 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.150150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.150983 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.150984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.151176 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.151962 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.151975 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.154267 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.156529 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.156549 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.156554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.156991 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.157201 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.157202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.157203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.157343 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.157615 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.157623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.157814 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.157926 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.158155 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260619 02:49:41.159068 590 heartbeater.cc:347] Connected to a master server at 127.26.98.62:38595
I20260619 02:49:41.159266 590 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:41.159513 590 heartbeater.cc:511] Master 127.26.98.62:38595 requested a full tablet report, sending...
I20260619 02:49:41.160158 32599 ts_manager.cc:194] Registered new tserver with Master: 8379e12719bc40969d759da2f7ab5df0 (127.26.98.3:40843)
I20260619 02:49:41.160691 32599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:47713
I20260619 02:49:41.168108 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/test-admin.keytab.
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.191250 27016 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 02:49:41.193383 597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:41.192437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.26.98.1:34001)
0619 02:49:41.192586 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.192592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.192611 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:49:41.192643 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.192645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.192662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.192762 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.193250 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59134: BlockingRecv error: recv got EOF from 127.0.0.1:59134 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
W20260619 02:49:41.193485 27016 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.194936 27016 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.196321 27016 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:41.207500 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.197111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.26.98.1:34001)
0619 02:49:41.197174 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.197176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.197255 (+ 79us) server_negotiation.cc:407] Connection header received
0619 02:49:41.197333 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.197334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.197348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.197400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.197866 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.198811 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.199794 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.200292 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.200402 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.203546 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.203563 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.203563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.203575 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.205573 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.206202 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.206208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.206209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.206344 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.206692 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.206693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.206694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.206995 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.207147 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.207307 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.207403 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 02:49:41.209632 597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:41.209071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.26.98.1:34001)
0619 02:49:41.209130 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.209132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.209140 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:41.209205 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.209206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.209220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.209267 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.209581 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59154: BlockingRecv error: recv got EOF from 127.0.0.1:59154 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 02:49:41.209879 27016 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc)
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.211133 27016 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.212126 27016 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:41.222097 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.212766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.26.98.1:34001)
0619 02:49:41.212832 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.212833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.212857 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:49:41.212929 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.212930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.212940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.212992 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.213431 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.214031 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.215029 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.215309 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.215415 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.218234 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.218236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.218237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.218247 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.220244 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.220911 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.220913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.220914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.221002 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.221386 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.221387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.221388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.221569 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.221671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.221842 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.221989 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 02:49:41.224413 597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:41.223713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.26.98.1:34001)
0619 02:49:41.223783 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.223785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.223878 (+ 93us) server_negotiation.cc:407] Connection header received
0619 02:49:41.223921 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.223922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.223931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.223980 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.224356 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59182: BlockingRecv error: recv got EOF from 127.0.0.1:59182 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 02:49:41.224529 27016 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837352638110-27016-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.225816 27016 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.226981 27016 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:49:41.236992 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.227827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.26.98.1:34001)
0619 02:49:41.227964 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.227967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.227988 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:49:41.228049 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.228050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.228065 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.228161 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.228678 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.229434 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.230550 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.230790 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.230867 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.233759 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.233761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.233762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.233777 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.235537 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.236060 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.236062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.236062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.236153 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.236397 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.236398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.236398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.236525 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.236669 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.236777 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.236880 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:49:41.237728 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32661
I20260619 02:49:41.243281 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 327
I20260619 02:49:41.247870 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 461
I20260619 02:49:41.252377 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 32567
2026-06-19T02:49:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2066 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[610](info): set up 2 sockets
Jun 19 02:49:41 dist-test-slave-0f9d krb5kdc[610](info): commencing operation
krb5kdc: starting...
W20260619 02:49:42.239354 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab.
Entry for principal oryx/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab.
I20260619 02:49:42.386650 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40659
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab
--principal=oryx/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:42.486215 623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:42.486546 623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:42.486575 623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:42.488232 623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:42.488329 623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:42.488344 623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:42.488358 623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:42.488368 623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:42.490465 623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab
--principal=oryx/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:42.490854 623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:42.491189 623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:42.497334 631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:42.497392 628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:42.497498 623 server_base.cc:1034] running on GCE node
W20260619 02:49:42.497597 629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:42.498009 623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:42.498412 623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:42.499599 623 hybrid_clock.cc:648] HybridClock initialized: now 1781837382499566 us; error 53 us; skew 500 ppm
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:42.502929 623 init.cc:377] Logged in from keytab as oryx/127.26.98.62@KRBTEST.COM (short username oryx)
I20260619 02:49:42.503510 623 webserver.cc:460] Webserver started at http://127.26.98.62:38139/ using document root <none> and password file <none>
I20260619 02:49:42.503767 623 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:42.503819 623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:42.503959 623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:42.505192 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "392149059ead4374b22e3c3bab158a95"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "7af8d51947310ae33dfc26fe3092725d"
server_key_iv: "13651d90a61e2a7c82e283cbca417945"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.505563 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "392149059ead4374b22e3c3bab158a95"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "7af8d51947310ae33dfc26fe3092725d"
server_key_iv: "13651d90a61e2a7c82e283cbca417945"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.507017 623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:42.507817 638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.508000 623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:42.508090 623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "392149059ead4374b22e3c3bab158a95"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "7af8d51947310ae33dfc26fe3092725d"
server_key_iv: "13651d90a61e2a7c82e283cbca417945"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.508157 623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:42.533845 623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:42.536813 623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:42.537006 623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:42.542073 623 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:40659
I20260619 02:49:42.542157 690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:40659 every 8 connection(s)
I20260619 02:49:42.542583 623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:42.543637 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 623
I20260619 02:49:42.543769 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:42.543880 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:42.543995 27016 external_mini_cluster.cc:1428] Setting key 50d2ff336d1b20c917d60cd41ab85877
I20260619 02:49:42.547787 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Bootstrap starting.
I20260619 02:49:42.548625 691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:42.548954 691 log.cc:824] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:42.549844 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: No bootstrap required, opened a new log
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:42.552551 691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } }
I20260619 02:49:42.552763 691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:42.552827 691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392149059ead4374b22e3c3bab158a95, State: Initialized, Role: FOLLOWER
I20260619 02:49:42.552996 691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } }
I20260619 02:49:42.553097 691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:42.553151 691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:42.553231 691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:42.553942 691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } }
I20260619 02:49:42.554069 691 leader_election.cc:302] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 392149059ead4374b22e3c3bab158a95; no voters:
I20260619 02:49:42.554277 691 leader_election.cc:288] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:42.554327 696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:42.554486 696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 LEADER]: Becoming Leader. State: Replica: 392149059ead4374b22e3c3bab158a95, State: Running, Role: LEADER
I20260619 02:49:42.554636 691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:42.554653 696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } }
I20260619 02:49:42.555119 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } }
I20260619 02:49:42.555266 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:42.556066 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 392149059ead4374b22e3c3bab158a95. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } }
I20260619 02:49:42.556278 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:42.556473 702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:42.556475 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.544689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.26.98.62:40659)
0619 02:49:42.544950 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.544961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.545001 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:42.545561 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.545573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.545652 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.545845 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:42.546567 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.548027 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.548913 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.549481 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.549602 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.552157 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:42.552164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:42.552165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:42.552185 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:42.554016 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.554487 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.554491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.554492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.554606 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.554852 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.554853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.554853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.555249 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:42.555403 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.555880 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.556054 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":131,"threads_started":1}
I20260619 02:49:42.557168 702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:42.559185 702 catalog_manager.cc:1347] Generated new cluster ID: f10cdd7eb55a4edfa2833f5025c627cf
I20260619 02:49:42.559263 702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:42.562682 702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:42.563288 702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:42.580734 702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Generated new TSK 0
I20260619 02:49:42.581032 702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
I20260619 02:49:42.620200 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:42.721171 717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:42.721484 717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:42.721514 717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:42.723114 717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:42.723196 717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:42.723234 717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:42.725227 717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:42.725577 717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:42.725880 717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:42.731978 723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:42.732210 725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:42.731995 722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:42.732329 717 server_base.cc:1034] running on GCE node
I20260619 02:49:42.732561 717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:42.732999 717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:42.734217 717 hybrid_clock.cc:648] HybridClock initialized: now 1781837382734195 us; error 50 us; skew 500 ppm
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:42.737679 717 init.cc:377] Logged in from keytab as oryx/127.26.98.1@KRBTEST.COM (short username oryx)
I20260619 02:49:42.738297 717 webserver.cc:460] Webserver started at http://127.26.98.1:35323/ using document root <none> and password file <none>
I20260619 02:49:42.738586 717 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:42.738649 717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:42.738806 717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:42.740254 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "3777adda73a14d3f8f79616234029d65"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "3a7424bc3b8f41dbe47c1958a5d69496"
server_key_iv: "ad1b3ec4d4c4ca93af9f9398c2e913c6"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.740669 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "3777adda73a14d3f8f79616234029d65"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "3a7424bc3b8f41dbe47c1958a5d69496"
server_key_iv: "ad1b3ec4d4c4ca93af9f9398c2e913c6"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.742358 717 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:49:42.743415 732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.743599 717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:42.743687 717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3777adda73a14d3f8f79616234029d65"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "3a7424bc3b8f41dbe47c1958a5d69496"
server_key_iv: "ad1b3ec4d4c4ca93af9f9398c2e913c6"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.743760 717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:42.755383 717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:42.758126 717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:42.758322 717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:42.758641 717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:42.759048 717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:42.759086 717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.759122 717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:42.759145 717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.766824 717 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36779
I20260619 02:49:42.766934 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36779 every 8 connection(s)
I20260619 02:49:42.767323 717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:42.777325 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 717
I20260619 02:49:42.777467 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:42.777837 27016 external_mini_cluster.cc:1428] Setting key 105e0e9611a56bf1ce5633728ffcbebc
I20260619 02:49:42.780166 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.768309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:41169 (local address 127.26.98.62:40659)
0619 02:49:42.768466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.768468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.768542 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:49:42.769207 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.769208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.769221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.769290 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:42.770453 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.771138 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.772680 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.772965 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.773104 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.776163 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:42.776166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:42.776166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:42.776177 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:42.777871 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.778552 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.778554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.778555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.778626 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.779098 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.779100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.779100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.779280 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:42.779372 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.779906 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.780045 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 02:49:42.780489 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.768204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:41169)
0619 02:49:42.768393 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:42.768441 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:42.769064 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:42.769365 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:42.769369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:42.770043 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:42.770316 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.770322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.771323 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.771325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.772500 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.772506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.773161 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.773162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.773347 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.774043 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:42.774054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:42.775931 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:42.778006 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:42.778023 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:42.778028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:42.778446 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:42.778785 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:42.778786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:42.778786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:42.779014 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:42.779395 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:42.779403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:42.779599 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 02:49:42.779848 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 02:49:42.780094 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260619 02:49:42.781322 846 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:42.781445 846 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:42.781690 846 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:42.782321 654 ts_manager.cc:194] Registered new tserver with Master: 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
I20260619 02:49:42.783100 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:41169
WARNING: no policy specified for oryx/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
I20260619 02:49:42.810956 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:42.917376 851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:42.917676 851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:42.917711 851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:42.919360 851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:42.919437 851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:42.919474 851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:42.921418 851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:42.921896 851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:42.922268 851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:42.928627 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:42.928611 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:42.928689 859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:42.929018 851 server_base.cc:1034] running on GCE node
I20260619 02:49:42.929286 851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:42.929579 851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:42.930760 851 hybrid_clock.cc:648] HybridClock initialized: now 1781837382930732 us; error 58 us; skew 500 ppm
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:42.934087 851 init.cc:377] Logged in from keytab as oryx/127.26.98.2@KRBTEST.COM (short username oryx)
I20260619 02:49:42.934736 851 webserver.cc:460] Webserver started at http://127.26.98.2:34129/ using document root <none> and password file <none>
I20260619 02:49:42.935041 851 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:42.935113 851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:42.935271 851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:42.936651 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "6ceec07f218e46b0bdad52c30c60430a"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "ff45cad86b2601e963f6a5b91b6aa86c"
server_key_iv: "843bfed021335f08e086fc090dcb9dc3"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.937062 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "6ceec07f218e46b0bdad52c30c60430a"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "ff45cad86b2601e963f6a5b91b6aa86c"
server_key_iv: "843bfed021335f08e086fc090dcb9dc3"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.938925 851 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:49:42.940299 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.940546 851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:42.940675 851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "6ceec07f218e46b0bdad52c30c60430a"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "ff45cad86b2601e963f6a5b91b6aa86c"
server_key_iv: "843bfed021335f08e086fc090dcb9dc3"
server_key_version: "encryptionkey@0"
I20260619 02:49:42.940784 851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:42.954702 851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:42.957720 851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:42.957895 851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:42.958370 851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:42.959017 851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:42.959079 851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.959106 851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:42.959122 851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:42.966209 851 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40415
I20260619 02:49:42.966295 979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40415 every 8 connection(s)
I20260619 02:49:42.966671 851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:42.968003 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 851
I20260619 02:49:42.968128 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:42.968348 27016 external_mini_cluster.cc:1428] Setting key d56fe0f2410c2bc349dc8f9331408246
Jun 19 02:49:42 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:49:42.979467 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.967609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:44611 (local address 127.26.98.62:40659)
0619 02:49:42.967700 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.967703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.967851 (+ 148us) server_negotiation.cc:407] Connection header received
0619 02:49:42.968661 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.968663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.968678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.968730 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:42.970160 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.970789 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.971944 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.972170 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.972232 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.975295 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:42.975298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:42.975298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:42.975309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:42.977086 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.977757 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.977760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.977760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.977855 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.978478 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.978480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.978480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.978657 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:42.978731 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.979196 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.979306 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:49:42.979676 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.967524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:44611)
0619 02:49:42.967701 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:42.967745 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:42.968526 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:42.968826 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:42.968830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:42.969631 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:42.969988 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.969996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.970934 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.970936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.971850 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.971853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.972274 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.972276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.972405 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.973054 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:42.973065 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:42.975161 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:42.977242 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:42.977256 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:42.977260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:42.977608 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:42.978015 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:42.978018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:42.978019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:42.978265 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:42.978754 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:42.978762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:42.979005 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 02:49:42.979140 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:49:42.979391 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260619 02:49:42.980268 980 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:42.980391 980 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:42.980620 980 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:42.981081 654 ts_manager.cc:194] Registered new tserver with Master: 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.3@KRBTEST.COM" created.
I20260619 02:49:42.981511 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:44611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
I20260619 02:49:42.998219 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.095722 985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.096014 985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.096045 985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.097467 985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.097523 985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.097556 985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:43.099403 985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:43.099690 985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.099937 985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.105536 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.105536 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.105536 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.105989 985 server_base.cc:1034] running on GCE node
I20260619 02:49:43.106251 985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.106547 985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.107718 985 hybrid_clock.cc:648] HybridClock initialized: now 1781837383107701 us; error 30 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.111512 985 init.cc:377] Logged in from keytab as oryx/127.26.98.3@KRBTEST.COM (short username oryx)
I20260619 02:49:43.112154 985 webserver.cc:460] Webserver started at http://127.26.98.3:44999/ using document root <none> and password file <none>
I20260619 02:49:43.112396 985 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.112452 985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.112602 985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.113981 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "62d104c829684c7793489e48278a1018"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "d53a951db07243ce563a8ad47246597f"
server_key_iv: "ca06de13b2a72cd591723fe9a7e73f83"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.114485 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "62d104c829684c7793489e48278a1018"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "d53a951db07243ce563a8ad47246597f"
server_key_iv: "ca06de13b2a72cd591723fe9a7e73f83"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.116396 985 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:49:43.117470 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.117697 985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:43.117806 985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "62d104c829684c7793489e48278a1018"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "d53a951db07243ce563a8ad47246597f"
server_key_iv: "ca06de13b2a72cd591723fe9a7e73f83"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.117890 985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.154031 985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.157191 985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.157366 985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.157747 985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:43.158176 985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:43.158218 985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.158252 985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:43.158273 985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.165638 985 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:37185
I20260619 02:49:43.165697 1113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:37185 every 8 connection(s)
I20260619 02:49:43.166121 985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:43.175767 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 985
I20260619 02:49:43.175882 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:43.176115 27016 external_mini_cluster.cc:1428] Setting key ff10bf379a5869e47c10a0fe586c7355
I20260619 02:49:43.178476 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.167069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:44127 (local address 127.26.98.62:40659)
0619 02:49:43.167171 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.167174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.167373 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:49:43.168166 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.168169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.168186 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.168279 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.169541 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.170097 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.171480 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.171697 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.171779 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.174807 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.174809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.174810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.174820 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.176439 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.177215 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.177217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.177218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.177286 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.177670 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.177671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.177672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.177822 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.177890 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.178284 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.178390 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:49:43.178727 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.167043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:44127)
0619 02:49:43.167224 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.167271 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.168011 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.168379 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.168384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.169147 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.169408 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.169414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.170207 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.170209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.171341 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.171346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.171805 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.171806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.171957 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.172629 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.172640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.174655 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.176550 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.176569 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.176573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.177013 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.177373 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.177374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.177374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.177579 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.177911 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.177918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.178109 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.178222 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.178404 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260619 02:49:43.179248 1114 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:43.179378 1114 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:43.179615 1114 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:43.180035 654 ts_manager.cc:194] Registered new tserver with Master: 62d104c829684c7793489e48278a1018 (127.26.98.3:37185)
I20260619 02:49:43.180421 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:44127
I20260619 02:49:43.188494 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.26.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.61.keytab.
Entry for principal oryx/127.26.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.61.keytab.
I20260619 02:49:43.218696 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:44427
--webserver_interface=127.26.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.61.keytab
--principal=oryx/127.26.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:40659,127.26.98.61:44427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.319010 1119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.319306 1119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.319332 1119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.320834 1119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:43.320917 1119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.320931 1119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.320945 1119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:43.320956 1119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:43.323920 1119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:40659,127.26.98.61:44427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.61.keytab
--principal=oryx/127.26.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
W20260619 02:49:43.324127 1119 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.26.98.62:40659,127.26.98.61:44427'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:49:43.324345 1119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.324647 1119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.330717 1127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.330717 1125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.330956 1124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.331303 1119 server_base.cc:1034] running on GCE node
I20260619 02:49:43.331704 1119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.332141 1119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.333351 1119 hybrid_clock.cc:648] HybridClock initialized: now 1781837383333323 us; error 43 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.336894 1119 init.cc:377] Logged in from keytab as oryx/127.26.98.61@KRBTEST.COM (short username oryx)
I20260619 02:49:43.337524 1119 webserver.cc:460] Webserver started at http://127.26.98.61:37205/ using document root <none> and password file <none>
I20260619 02:49:43.337775 1119 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.337831 1119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.337980 1119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.339493 1119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/instance:
uuid: "de9778a2c2894229b7e747b8a9b7f42c"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "4c541b94c659fdab519afa652a7ffe0c"
server_key_iv: "e6cb4f714569952b420efd0a6c8256c5"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.340058 1119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal/instance:
uuid: "de9778a2c2894229b7e747b8a9b7f42c"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "4c541b94c659fdab519afa652a7ffe0c"
server_key_iv: "e6cb4f714569952b420efd0a6c8256c5"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.341650 1119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:43.342602 1134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.342846 1119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:43.343075 1119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "de9778a2c2894229b7e747b8a9b7f42c"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "4c541b94c659fdab519afa652a7ffe0c"
server_key_iv: "e6cb4f714569952b420efd0a6c8256c5"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.343297 1119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.356546 1119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.359165 1119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.359344 1119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.364245 1119 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.61:44427
I20260619 02:49:43.364331 1186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.61:44427 every 8 connection(s)
I20260619 02:49:43.364737 1119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
I20260619 02:49:43.365494 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 1119
I20260619 02:49:43.366119 1187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.368995 1187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.61@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:43.378762 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.366329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.26.98.61:44427)
0619 02:49:43.366530 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.366542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.366589 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:49:43.367294 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.367306 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.367389 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.367625 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.368376 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.369795 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.370839 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.371528 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.371629 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.375020 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.375028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.375028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.375037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.376976 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.377372 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.377375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.377376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.377475 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.377660 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.377661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.377661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.377974 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.378162 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.378252 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.378429 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260619 02:49:43.379431 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.370227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.26.98.62:40659)
0619 02:49:43.370312 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.370314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.370530 (+ 216us) server_negotiation.cc:407] Connection header received
0619 02:49:43.370635 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.370637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.370651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.370703 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.371468 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.372136 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.373179 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.373521 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.373622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.376649 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.376652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.376653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.376663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.377982 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.378386 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.378387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.378388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.378462 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.378690 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.378690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.378691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.378870 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.378995 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.379234 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.379337 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:43.379458 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.370077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39334)
0619 02:49:43.370409 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.370453 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.370542 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.370739 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.370743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.371228 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.371319 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.371321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.372283 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.372284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.373042 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.373045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.373799 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.373801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.373926 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.374548 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.374561 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.376515 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.378073 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.378080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.378083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.378299 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.378542 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.378543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.378543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.378617 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.378987 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.378992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.379054 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.379182 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.379376 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":173,"threads_started":1}
I20260619 02:49:43.380097 1187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.61@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
I20260619 02:49:43.388501 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.380469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.26.98.61:44427)
0619 02:49:43.380527 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.380530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.380608 (+ 78us) server_negotiation.cc:407] Connection header received
0619 02:49:43.380703 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.380704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.380713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.380758 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.381106 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.381616 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.382266 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.382460 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.382519 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.385310 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.385312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.385313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.385321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.386913 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.387414 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.387415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.387415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.387477 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.387738 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.387739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.387740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.387881 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.387949 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.388272 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.388416 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":26}
I20260619 02:49:43.388506 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.380430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44154)
0619 02:49:43.380524 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.380542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.380637 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.380766 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.380767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.380967 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.381036 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.381038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.381685 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.381686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.382188 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.382190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.382522 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.382522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.382578 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.383197 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.383204 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.385122 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.387039 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.387040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.387041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.387261 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.387568 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.387570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.387570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.387646 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.388024 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.388027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.388103 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.388208 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.388356 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:49:43.390574 1187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Bootstrap starting.
I20260619 02:49:43.391800 1187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.392294 1187 log.cc:824] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.393282 1187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: No bootstrap required, opened a new log
I20260619 02:49:43.395339 1187 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } }
I20260619 02:49:43.395534 1187 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.395581 1187 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9778a2c2894229b7e747b8a9b7f42c, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.395731 1187 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } }
I20260619 02:49:43.396034 1193 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } } }
I20260619 02:49:43.396129 1187 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.396142 1193 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:49:43.398204 1204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:43.398324 1204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:43.400919 1119 master_runner.cc:428] Detected that this master de9778a2c2894229b7e747b8a9b7f42c is joining an existing cluster
I20260619 02:49:43.401016 1119 master_runner.cc:432] Initiating AddMaster RPC to add 127.26.98.61:44427
I20260619 02:49:43.409399 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.402754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.26.98.62:40659)
0619 02:49:43.402818 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.402821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.402991 (+ 170us) server_negotiation.cc:407] Connection header received
0619 02:49:43.403126 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.403128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.403142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.403177 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.403792 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.404542 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.405380 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.405579 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.405640 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.406791 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.406793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.406794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.406802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.408242 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.408629 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.408630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.408631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.408694 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.408851 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.408852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.408852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.408980 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.409037 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.409198 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.409283 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:43.409411 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.402647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39338)
0619 02:49:43.402855 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.402869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.403027 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.403255 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.403255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.403590 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.403673 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.403675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.404636 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.404637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.405292 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.405293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.405653 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.405653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.405726 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.406230 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.406234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.406688 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.408331 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.408332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.408333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.408555 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.408747 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.408747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.408748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.408801 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.409031 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.409032 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.409107 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.409163 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.409290 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":36,"thread_start_us":112,"threads_started":1}
I20260619 02:49:43.410176 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.403064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.26.98.61:44427)
0619 02:49:43.403119 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.403121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.403152 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:43.403208 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.403209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.403218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.403272 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.403733 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.404519 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.405664 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.405944 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.405982 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.407416 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.407417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.407418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.407424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.409020 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.409411 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.409411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.409412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.409479 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.409720 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.409720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.409721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.409829 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.409867 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.410014 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.410098 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:43.410465 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.402946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44164)
0619 02:49:43.403083 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.403093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.403156 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.403294 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.403294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.403546 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.403652 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.403655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.404629 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.404630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.405567 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.405570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.406064 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.406064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.406129 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.406475 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.406480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.407324 (+ 844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.409121 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.409122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.409123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.409343 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.409599 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.409600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.409600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.409660 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.409901 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.409902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.409936 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.410285 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.410379 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":39,"thread_start_us":70,"threads_started":1}
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.62@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
I20260619 02:49:43.420675 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.412341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.26.98.61:44427)
0619 02:49:43.412383 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.412385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.412549 (+ 164us) server_negotiation.cc:407] Connection header received
0619 02:49:43.412767 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.412768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.412780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.412841 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.413338 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.413838 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.414636 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.414842 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.414918 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.417595 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.417598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.417598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.417609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.419228 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.419736 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.419738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.419738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.419809 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.420034 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.420035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.420035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.420205 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.420277 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.420454 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.420555 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:43.420722 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.412237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44176)
0619 02:49:43.412441 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.412482 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.412649 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.412852 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.412856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.413185 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.413265 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.413268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.413909 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.413910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.414559 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.414561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.414928 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.414929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.415016 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.415630 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.415635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.417471 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.419353 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.419366 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.419369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.419653 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.419904 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.419904 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.419905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.419974 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.420270 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.420279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.420340 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.420414 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.420601 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260619 02:49:43.421766 654 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.26.98.61:44427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c"
member_type: NON_VOTER
last_known_addr {
host: "127.26.98.61"
port: 44427
}
attrs {
promote: true
}
}
dest_uuid: "392149059ead4374b22e3c3bab158a95"
cas_config_opid_index: -1
I20260619 02:49:43.422184 654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } }
I20260619 02:49:43.422737 1214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } }
I20260619 02:49:43.423240 1161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:43.423823 1214 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.26.98.61:44427
I20260619 02:49:43.424093 1119 master.cc:537] Master@127.26.98.61:44427 shutting down...
I20260619 02:49:43.424053 1213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } }
I20260619 02:49:43.424144 1213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.424328 1161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:49:43.424741 1213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:49:43.425655 641 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 02:49:43.426296 641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 -> Peer de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61:44427): Couldn't send request to peer de9778a2c2894229b7e747b8a9b7f42c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:49:43.428215 1119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:43.428361 1119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:43.428426 1119 tablet_replica.cc:330] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: stopping tablet replica
I20260619 02:49:43.442432 1119 master.cc:552] Master@127.26.98.61:44427 shutdown complete.
I20260619 02:49:43.442507 1119 master_runner.cc:305] Clearing existing system tablet
I20260619 02:49:43.442626 1119 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.442696 1119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.443635 1119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:43.444458 1218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.444675 1119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.444780 1119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:43.444823 1119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "de9778a2c2894229b7e747b8a9b7f42c"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "4c541b94c659fdab519afa652a7ffe0c"
server_key_iv: "e6cb4f714569952b420efd0a6c8256c5"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.445672 1119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:43.447490 1119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:49:43.447592 1119 log.cc:1196] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:49:43.447841 1119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Deleting consensus metadata
I20260619 02:49:43.447954 1119 master_runner.cc:315] Copying system tablet from 127.26.98.62:40659
I20260619 02:49:43.448632 1119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: tablet copy: Beginning tablet copy session from remote peer at address 127.26.98.62:40659
I20260619 02:49:43.455369 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.449046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39342)
0619 02:49:43.449207 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.449220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.449305 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.449437 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.449438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.449640 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.449706 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.449708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.450423 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.450424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.451070 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.451073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.451418 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.451419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.451482 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.452039 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.452044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.452420 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.454025 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.454027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.454027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.454246 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.454442 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.454443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.454443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.454507 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.454813 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.454814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.454928 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.455017 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.455225 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260619 02:49:43.455386 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.449141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.26.98.62:40659)
0619 02:49:43.449209 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.449211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.449287 (+ 76us) server_negotiation.cc:407] Connection header received
0619 02:49:43.449375 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.449376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.449388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.449433 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.449789 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.450340 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.451149 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.451355 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.451416 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.452542 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.452545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.452545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.452553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.453922 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.454314 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.454315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.454315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.454383 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.454559 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.454560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.454560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.454705 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.454781 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.455058 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.455257 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:43.456291 675 tablet_copy_service.cc:140] P 392149059ead4374b22e3c3bab158a95: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer de9778a2c2894229b7e747b8a9b7f42c ({username='oryx', principal='oryx/127.26.98.61@KRBTEST.COM'} at 127.0.0.1:39342)
I20260619 02:49:43.456439 675 tablet_copy_service.cc:161] P 392149059ead4374b22e3c3bab158a95: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer de9778a2c2894229b7e747b8a9b7f42c at {username='oryx', principal='oryx/127.26.98.61@KRBTEST.COM'} at 127.0.0.1:39342: session id = de9778a2c2894229b7e747b8a9b7f42c-00000000000000000000000000000000
I20260619 02:49:43.457540 675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:49:43.458514 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:49:43.460199 1119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: tablet copy: Starting download of 0 data blocks...
I20260619 02:49:43.460412 1119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: tablet copy: Starting download of 1 WAL segments...
I20260619 02:49:43.463241 1119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:49:43.465201 675 tablet_copy_service.cc:342] P 392149059ead4374b22e3c3bab158a95: Request end of tablet copy session de9778a2c2894229b7e747b8a9b7f42c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.26.98.61@KRBTEST.COM'} at 127.0.0.1:39342
I20260619 02:49:43.465301 675 tablet_copy_service.cc:433] P 392149059ead4374b22e3c3bab158a95: ending tablet copy session de9778a2c2894229b7e747b8a9b7f42c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer de9778a2c2894229b7e747b8a9b7f42c
I20260619 02:49:43.469242 1119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.471012 1226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.471014 1225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.472498 1119 server_base.cc:1034] running on GCE node
W20260619 02:49:43.475080 1228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.475364 1119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.475584 1119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.476730 1119 hybrid_clock.cc:648] HybridClock initialized: now 1781837383476720 us; error 46 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.478699 1119 init.cc:377] Logged in from keytab as oryx/127.26.98.61@KRBTEST.COM (short username oryx)
I20260619 02:49:43.479156 1119 webserver.cc:460] Webserver started at http://127.26.98.61:43327/ using document root <none> and password file <none>
I20260619 02:49:43.479286 1119 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.479328 1119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.480067 1119 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:43.481099 1235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.481290 1119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:43.481345 1119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "de9778a2c2894229b7e747b8a9b7f42c"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "4c541b94c659fdab519afa652a7ffe0c"
server_key_iv: "e6cb4f714569952b420efd0a6c8256c5"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.481423 1119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.501634 1119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.503930 1119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.507790 1119 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.61:44427
I20260619 02:49:43.507938 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.61:44427 every 8 connection(s)
I20260619 02:49:43.508261 1119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
I20260619 02:49:43.509608 1288 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:49:43.510618 1288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Bootstrap starting.
I20260619 02:49:43.514825 1288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:49:43.515192 1288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: Bootstrap complete.
I20260619 02:49:43.515478 1288 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } }
I20260619 02:49:43.515589 1288 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9778a2c2894229b7e747b8a9b7f42c, State: Initialized, Role: LEARNER
I20260619 02:49:43.515633 1288 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } }
I20260619 02:49:43.515936 1288 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.515910 1291 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } }
I20260619 02:49:43.516055 1291 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: LEARNER
I20260619 02:49:43.518111 1302 catalog_manager.cc:1259] Loaded cluster ID: f10cdd7eb55a4edfa2833f5025c627cf
I20260619 02:49:43.518260 1302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: loading cluster ID for follower catalog manager: success
I20260619 02:49:43.519399 1302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: acquiring CA information for follower catalog manager: success
I20260619 02:49:43.519797 1302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:49:43.539016 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.531618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.26.98.61:44427)
0619 02:49:43.531779 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.531781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.531790 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:43.531817 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.531818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.531829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.531901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.532499 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.533373 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.534250 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.534500 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.534581 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.536135 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.536137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.536137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.536149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.537699 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.538094 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.538095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.538096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.538156 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.538324 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.538325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.538325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.538464 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.538595 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.538728 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.538841 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.26.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.60.keytab.
Entry for principal oryx/127.26.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.60.keytab.
I20260619 02:49:43.569872 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:45421
--webserver_interface=127.26.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.60.keytab
--principal=oryx/127.26.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.671044 1306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.671352 1306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.671380 1306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.672808 1306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:43.672869 1306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.672884 1306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.672897 1306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:43.672909 1306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:43.674803 1306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.60.keytab
--principal=oryx/127.26.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:43.675201 1306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.675467 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.681177 1312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.681171 1311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.681175 1314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.681622 1306 server_base.cc:1034] running on GCE node
I20260619 02:49:43.681903 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.682314 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.683503 1306 hybrid_clock.cc:648] HybridClock initialized: now 1781837383683436 us; error 77 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.687446 1306 init.cc:377] Logged in from keytab as oryx/127.26.98.60@KRBTEST.COM (short username oryx)
I20260619 02:49:43.688081 1306 webserver.cc:460] Webserver started at http://127.26.98.60:46513/ using document root <none> and password file <none>
I20260619 02:49:43.688361 1306 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.688426 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.688585 1306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.689966 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/instance:
uuid: "28b079c8bda844f79b2ac1c702946dc3"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "6a59c4a33d5dcf26445bd897bf51d179"
server_key_iv: "aa720127402d3676e0014f72190c4d59"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.690485 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal/instance:
uuid: "28b079c8bda844f79b2ac1c702946dc3"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "6a59c4a33d5dcf26445bd897bf51d179"
server_key_iv: "aa720127402d3676e0014f72190c4d59"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.692242 1306 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:49:43.693271 1321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.693517 1306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:43.693631 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "28b079c8bda844f79b2ac1c702946dc3"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "6a59c4a33d5dcf26445bd897bf51d179"
server_key_iv: "aa720127402d3676e0014f72190c4d59"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.693718 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.715507 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.718568 1306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.718761 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.724201 1306 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.60:45421
I20260619 02:49:43.724279 1373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.60:45421 every 8 connection(s)
I20260619 02:49:43.724706 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
I20260619 02:49:43.726168 1374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.726852 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 1306
I20260619 02:49:43.728631 1374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.60@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
I20260619 02:49:43.739596 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.729740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.26.98.62:40659)
0619 02:49:43.729804 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.729806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.729896 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:49:43.729962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.729963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.729975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.730040 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.730609 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.731296 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.732451 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.732735 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.732825 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.736164 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.736167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.736168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.736181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.737811 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.738349 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.738351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.738351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.738412 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.738712 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.738714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.738714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.738951 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.739049 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.739414 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.739499 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:49:43.739850 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.729554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39344)
0619 02:49:43.729773 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.729823 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.729908 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.730070 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.730072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.730382 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.730505 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.730510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.731418 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.731420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.732359 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.732362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.732897 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.732899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.732985 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.733741 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.733756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.735992 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.737912 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.737919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.737924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.738269 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.738472 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.738473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.738473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.738622 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.739033 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.739037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.739247 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.739353 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.739552 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":148,"threads_started":1}
I20260619 02:49:43.740267 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.727728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.26.98.60:45421)
0619 02:49:43.727973 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.727983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.728014 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:43.728560 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.728569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.728636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.728858 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.729653 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.730971 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.732040 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.732599 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.732731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.736747 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.736761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.736761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.736771 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.738580 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.739098 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.739101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.739102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.739162 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.739392 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.739393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.739393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.739601 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.739753 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.740082 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.740167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":120,"threads_started":1}
I20260619 02:49:43.740516 1374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.60@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
I20260619 02:49:43.748358 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.740923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.26.98.61:44427)
0619 02:49:43.741001 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.741004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.741013 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:43.741128 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.741129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.741144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.741196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.741485 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.742030 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.742832 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.743128 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.743189 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.745614 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.745618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.745618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.745628 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.747193 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.747658 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.747660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.747660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.747744 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.747898 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.747898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.747899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.748028 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.748079 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.748185 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.748275 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:49:43.748368 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.740852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44192)
0619 02:49:43.740970 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.740980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.741029 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.741216 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.741217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.741334 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.741396 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.741397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.742152 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.742154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.742736 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.742738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.743233 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.743234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.743317 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.743922 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.743929 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.745463 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.747328 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.747330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.747331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.747565 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.747804 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.747805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.747805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.747860 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.748072 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.748073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.748118 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.748164 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.748287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 02:49:43.748867 1374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.60@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
I20260619 02:49:43.756570 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.749217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.26.98.60:45421)
0619 02:49:43.749262 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.749264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.749289 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:49:43.749336 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.749337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.749346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.749389 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.749632 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.750153 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.750796 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.751087 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.751154 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.753598 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.753602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.753602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.753615 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.755384 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.755732 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.755733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.755733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.755784 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.755990 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.755992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.755992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.756156 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.756225 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.756349 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.756446 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
I20260619 02:49:43.756667 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.749182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58134)
0619 02:49:43.749230 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.749243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.749283 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.749390 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.749391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.749528 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.749571 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.749573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.750230 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.750231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.750721 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.750723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.751149 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.751150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.751195 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.751775 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.751780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.753394 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.755488 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.755489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.755490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.755667 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.755834 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.755835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.755836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.755909 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.756227 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.756227 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.756271 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.756319 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.756488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":65}
I20260619 02:49:43.758355 1374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Bootstrap starting.
I20260619 02:49:43.759370 1374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.759657 1374 log.cc:824] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.760484 1374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: No bootstrap required, opened a new log
I20260619 02:49:43.762156 1374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } }
I20260619 02:49:43.762341 1374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.762377 1374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b079c8bda844f79b2ac1c702946dc3, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.762526 1374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } }
I20260619 02:49:43.763005 1379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } } }
I20260619 02:49:43.763101 1374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.763119 1379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:49:43.765049 1390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:43.765129 1390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:43.767874 1306 master_runner.cc:428] Detected that this master 28b079c8bda844f79b2ac1c702946dc3 is joining an existing cluster
I20260619 02:49:43.767951 1306 master_runner.cc:432] Initiating AddMaster RPC to add 127.26.98.60:45421
I20260619 02:49:43.775126 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.769209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.26.98.60:45421)
0619 02:49:43.769259 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.769261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.769423 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:49:43.769478 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.769479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.769489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.769541 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.769934 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.770435 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.771404 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.771600 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.771663 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.772746 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.772748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.772748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.772756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.774047 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.774358 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.774360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.774360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.774409 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.774559 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.774560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.774560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.774670 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.774730 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.774848 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.775045 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":22}
I20260619 02:49:43.775130 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.769197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58136)
0619 02:49:43.769358 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.769367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.769431 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.769563 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.769564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.769764 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.769854 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.769855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.770506 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.770506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.771338 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.771341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.771678 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.771679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.771731 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.772305 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.772310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.772666 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.774120 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.774121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.774122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.774301 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.774458 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.774458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.774458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.774514 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.774725 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.774726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.774764 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.774825 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.774988 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":36,"thread_start_us":109,"threads_started":1}
I20260619 02:49:43.784278 846 heartbeater.cc:503] Master 127.26.98.62:40659 was elected leader, sending a full tablet report...
I20260619 02:49:43.785866 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.779231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.26.98.62:40659)
0619 02:49:43.779291 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.779293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.779302 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:43.779453 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.779454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.779471 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.779528 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.779934 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.780469 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.781252 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.781525 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.781600 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.782690 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.782692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.782692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.782702 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.784494 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785011 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.785013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.785013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.785078 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785287 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.785287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.785288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.785438 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.785485 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.785670 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.785763 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:43.785889 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.779123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39352)
0619 02:49:43.779198 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.779213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.779368 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.779559 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.779561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.779796 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.779858 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.779860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.780555 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.780556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.781179 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.781182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.781596 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.781598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.781672 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.782152 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.782157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.782608 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.784715 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.784717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.784717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.784934 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.785157 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.785158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.785220 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.785528 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.785529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.785574 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.785628 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.785763 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:49:43.786526 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.779316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.26.98.61:44427)
0619 02:49:43.779387 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.779389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.779523 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:49:43.779604 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.779605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.779616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.779644 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.780104 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.780891 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.782046 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.782364 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.782406 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.783535 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.783544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.783544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.783553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.785250 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785656 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.785657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.785657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.785717 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785961 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.785962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.785963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.786107 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.786147 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.786351 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.786437 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:49:43.787329 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.779212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44202)
0619 02:49:43.779451 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.779460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.779525 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.779712 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.779713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.779953 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.780022 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.780024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.780987 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.780987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.781952 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.781955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.782489 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.782490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.782573 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.783016 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.783020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.783447 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.785356 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.785357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.785576 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.785796 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.785797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.785797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.785868 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.786188 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.786189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.786234 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.786405 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.786523 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":191,"threads_started":1}
I20260619 02:49:43.791868 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 717
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837382, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.62@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
I20260619 02:49:43.799304 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:36779
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:40659
--webserver_port=35323
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:49:43.822322 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.788681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.26.98.60:45421)
0619 02:49:43.788727 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.788728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.788794 (+ 66us) server_negotiation.cc:407] Connection header received
0619 02:49:43.788888 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.788889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.788901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.788958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.789315 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.789820 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.790855 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.791095 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.791179 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.793717 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.793719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.793720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.793729 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.820831 (+ 27102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.821416 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.821419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.821419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.821490 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.821693 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.821693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.821694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.821866 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.821933 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.822116 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.822203 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:43.822322 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.788607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58150)
0619 02:49:43.788736 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.788749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.788812 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.788969 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.788969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.789172 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.789238 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.789241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.789886 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.789887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.790747 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.790750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.791163 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.791164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.791242 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.791820 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.791826 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.793604 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.820956 (+ 27352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.820958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.820959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.821307 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.821554 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.821555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.821556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.821628 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.821927 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.821928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.821988 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.822052 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.822212 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 02:49:43.823288 654 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.26.98.60:45421: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3"
member_type: NON_VOTER
last_known_addr {
host: "127.26.98.60"
port: 45421
}
attrs {
promote: true
}
}
dest_uuid: "392149059ead4374b22e3c3bab158a95"
cas_config_opid_index: 5
I20260619 02:49:43.823601 654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } }
I20260619 02:49:43.823992 1214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.824906 1214 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.26.98.60:45421
I20260619 02:49:43.824985 1214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.825047 1214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.825592 1306 master.cc:537] Master@127.26.98.60:45421 shutting down...
I20260619 02:49:43.825829 1348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:43.827090 1348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:49:43.827541 1213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:49:43.828749 639 consensus_peers.cc:487] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 -> Peer 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60:45421): Couldn't send request to peer 28b079c8bda844f79b2ac1c702946dc3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:49:43.829900 1306 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:43.830127 1306 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:43.830288 1306 tablet_replica.cc:330] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: stopping tablet replica
I20260619 02:49:43.833374 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.825281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:44216)
0619 02:49:43.829266 (+ 3985us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.829284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.829347 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.829711 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.829713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.829716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:43.829854 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.829857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.830832 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.830833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.832191 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.832195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.833068 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.833069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.833137 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.833170 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.833227 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.833291 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3956}
I20260619 02:49:43.833654 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.825359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.26.98.61:44427)
0619 02:49:43.825409 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.825412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.829409 (+ 3997us) server_negotiation.cc:407] Connection header received
0619 02:49:43.829449 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.829450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.829482 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.829540 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:43.829939 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.830731 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.832286 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.832887 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.832960 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.833031 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.833536 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.833586 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:43.833948 1262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 LEARNER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:49:43.834200 1213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:49:43.834501 1306 master.cc:552] Master@127.26.98.60:45421 shutdown complete.
I20260619 02:49:43.834537 1306 master_runner.cc:305] Clearing existing system tablet
I20260619 02:49:43.834666 1306 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.834709 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.834957 1262 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.835515 1306 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.835976 1401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.836063 1306 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.836102 1306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.836119 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "28b079c8bda844f79b2ac1c702946dc3"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "6a59c4a33d5dcf26445bd897bf51d179"
server_key_iv: "aa720127402d3676e0014f72190c4d59"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.836618 1291 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 392149059ead4374b22e3c3bab158a95. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.836720 1291 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: LEARNER
I20260619 02:49:43.836759 1291 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: NON_VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: true } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.836793 1291 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: LEARNER
I20260619 02:49:43.837033 1306 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:43.837764 1213 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: attempting to promote NON_VOTER de9778a2c2894229b7e747b8a9b7f42c to VOTER
I20260619 02:49:43.838162 1213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } }
I20260619 02:49:43.839114 1262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 LEARNER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 02:49:43.839644 1214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 02:49:43.839426 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.839261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58160)
0619 02:49:43.839318 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.839383 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:49:43.840526 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.840403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58170)
0619 02:49:43.840454 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.840486 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:49:43.840615 639 consensus_peers.cc:487] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 -> Peer 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60:45421): Couldn't send request to peer 28b079c8bda844f79b2ac1c702946dc3. Status: Network error: Client connection negotiation failed: client connection to 127.26.98.60:45421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:49:43.840828 1214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.842175 1262 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.843531 1306 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:49:43.843645 1306 log.cc:1196] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:49:43.844123 1306 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Deleting consensus metadata
I20260619 02:49:43.844420 1306 master_runner.cc:315] Copying system tablet from 127.26.98.62:40659
I20260619 02:49:43.844533 1214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.844655 1214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.844710 1214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.844748 1214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.844962 1402 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.845100 1402 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:49:43.845827 1306 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: tablet copy: Beginning tablet copy session from remote peer at address 127.26.98.62:40659
I20260619 02:49:43.852768 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.846306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:39366)
0619 02:49:43.846497 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.846508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.846592 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.846732 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.846733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.847106 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.847183 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.847186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.847913 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.847915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.848656 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.848660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.849069 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.849070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.849161 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.849694 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.849699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.850073 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.851677 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.851678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.851679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.851874 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.852072 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.852073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.852074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.852144 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.852385 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.852386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.852442 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.852506 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.852649 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260619 02:49:43.852764 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.846363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.26.98.62:40659)
0619 02:49:43.846419 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.846435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.846589 (+ 154us) server_negotiation.cc:407] Connection header received
0619 02:49:43.846664 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.846665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.846677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.846727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.847267 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.847827 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.848777 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.848999 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.849061 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.850187 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.850189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.850190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.850199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.851575 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.851950 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.851951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.851952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.852013 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.852196 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.852197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.852197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.852330 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.852389 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.852537 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.852646 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:43.853403 675 tablet_copy_service.cc:140] P 392149059ead4374b22e3c3bab158a95: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 28b079c8bda844f79b2ac1c702946dc3 ({username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:39366)
I20260619 02:49:43.853468 675 tablet_copy_service.cc:161] P 392149059ead4374b22e3c3bab158a95: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 28b079c8bda844f79b2ac1c702946dc3 at {username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:39366: session id = 28b079c8bda844f79b2ac1c702946dc3-00000000000000000000000000000000
I20260619 02:49:43.854023 675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:49:43.854610 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:49:43.856027 1306 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: tablet copy: Starting download of 0 data blocks...
I20260619 02:49:43.856205 1306 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: tablet copy: Starting download of 1 WAL segments...
I20260619 02:49:43.857147 1306 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:49:43.858243 675 tablet_copy_service.cc:342] P 392149059ead4374b22e3c3bab158a95: Request end of tablet copy session 28b079c8bda844f79b2ac1c702946dc3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:39366
I20260619 02:49:43.858312 675 tablet_copy_service.cc:433] P 392149059ead4374b22e3c3bab158a95: ending tablet copy session 28b079c8bda844f79b2ac1c702946dc3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 28b079c8bda844f79b2ac1c702946dc3
I20260619 02:49:43.861980 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.863504 1411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.863950 1412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.864152 1414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.864401 1306 server_base.cc:1034] running on GCE node
I20260619 02:49:43.864539 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.864718 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.865849 1306 hybrid_clock.cc:648] HybridClock initialized: now 1781837383865834 us; error 37 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.867465 1306 init.cc:377] Logged in from keytab as oryx/127.26.98.60@KRBTEST.COM (short username oryx)
I20260619 02:49:43.867892 1306 webserver.cc:460] Webserver started at http://127.26.98.60:42311/ using document root <none> and password file <none>
I20260619 02:49:43.868021 1306 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.868057 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.868749 1306 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:43.869338 1421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.869443 1306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.869484 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "28b079c8bda844f79b2ac1c702946dc3"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "6a59c4a33d5dcf26445bd897bf51d179"
server_key_iv: "aa720127402d3676e0014f72190c4d59"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.869561 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.883071 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.884786 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.888821 1306 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.60:45421
I20260619 02:49:43.889294 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
I20260619 02:49:43.891235 1473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.60:45421 every 8 connection(s)
I20260619 02:49:43.895641 1474 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:49:43.896190 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Bootstrap starting.
I20260619 02:49:43.899277 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:49:43.899585 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Bootstrap complete.
I20260619 02:49:43.899857 1474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } }
I20260619 02:49:43.899948 1474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 28b079c8bda844f79b2ac1c702946dc3, State: Initialized, Role: LEARNER
I20260619 02:49:43.899991 1474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } }
I20260619 02:49:43.900302 1474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.900259 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:43.900507 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:49:43.902391 1488 catalog_manager.cc:1259] Loaded cluster ID: f10cdd7eb55a4edfa2833f5025c627cf
I20260619 02:49:43.902550 1488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: loading cluster ID for follower catalog manager: success
I20260619 02:49:43.904107 1488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: acquiring CA information for follower catalog manager: success
I20260619 02:49:43.904484 1488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:49:43.939481 1398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.939782 1398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.939815 1398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.941359 1398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.941442 1398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.941479 1398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:43.943557 1398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:36779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=35323
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:43.943920 1398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.944262 1398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.952687 1494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.952687 1496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.952965 1398 server_base.cc:1034] running on GCE node
W20260619 02:49:43.952689 1493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.953370 1398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.953765 1398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.954958 1398 hybrid_clock.cc:648] HybridClock initialized: now 1781837383954922 us; error 61 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.958811 1398 init.cc:377] Logged in from keytab as oryx/127.26.98.1@KRBTEST.COM (short username oryx)
I20260619 02:49:43.959580 1398 webserver.cc:460] Webserver started at http://127.26.98.1:35323/ using document root <none> and password file <none>
I20260619 02:49:43.959878 1398 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.959987 1398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.962363 1398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:43.963343 1503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.963557 1398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:43.963647 1398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3777adda73a14d3f8f79616234029d65"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "3a7424bc3b8f41dbe47c1958a5d69496"
server_key_iv: "ad1b3ec4d4c4ca93af9f9398c2e913c6"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.964110 1398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.975656 1398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.978699 1398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.978914 1398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.979291 1398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:43.979733 1398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:43.979787 1398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.979818 1398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:43.979831 1398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.982666 980 heartbeater.cc:503] Master 127.26.98.62:40659 was elected leader, sending a full tablet report...
I20260619 02:49:43.989158 1398 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36779
I20260619 02:49:43.989354 1616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36779 every 8 connection(s)
I20260619 02:49:43.989884 1398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
Jun 19 02:49:43 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
I20260619 02:49:43.999259 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 1398
I20260619 02:49:43.999444 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 851
I20260619 02:49:44.002601 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35311 (local address 127.26.98.62:40659)
0619 02:49:43.991300 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.991303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.991575 (+ 272us) server_negotiation.cc:407] Connection header received
0619 02:49:43.992318 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.992320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.992335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.992401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.993397 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.993956 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995441 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995720 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.998551 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.998554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.998554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.998566 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.000552 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.001323 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.001325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.001325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.001398 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.001706 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.001707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.001708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.001883 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.001946 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.002407 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.002512 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:44.003182 1622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:35311)
0619 02:49:43.991376 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.991449 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.992156 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.992512 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.992518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.992954 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.993280 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.993286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.994068 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.994069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995325 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.995328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995741 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995900 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.996665 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.996678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.998431 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.000808 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.000834 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.000839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.001213 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.001474 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.001475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.001475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.001635 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.001940 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.001948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.002163 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.002290 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.002520 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":25,"thread_start_us":141,"threads_started":1}
I20260619 02:49:44.003830 1618 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:44.003953 1618 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.004186 1618 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:44.004705 654 ts_manager.cc:194] Re-registered known tserver with Master: 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
I20260619 02:49:44.005177 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:35311
I20260619 02:49:44.007411 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:40415
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:40659
--webserver_port=34129
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:49:44.008455 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.26.98.1:51415)
0619 02:49:43.992871 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.992884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.992964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.993117 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.993118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.993290 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.993353 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.993354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.994273 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.994273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995325 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.995328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995806 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995904 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.996698 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.996702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.998788 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.007012 (+ 8224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.007013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.007014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.007289 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.007565 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.007566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.007566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.007638 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.008009 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.008011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.008068 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.008164 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.008318 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":11,"thread_start_us":69,"threads_started":1}
I20260619 02:49:44.008922 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:51415 (local address 127.26.98.61:44427)
0619 02:49:43.991654 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.991656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.992976 (+ 1320us) server_negotiation.cc:407] Connection header received
0619 02:49:43.993030 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.993031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.993045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.993098 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.993413 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.994185 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995398 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995696 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995748 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.998932 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.998934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.998935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.998943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.006901 (+ 7958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.007377 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.007379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.007379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.007476 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.007700 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.007701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.007701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.007888 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.007960 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.008335 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.008451 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:44.009716 1619 heartbeater.cc:347] Connected to a master server at 127.26.98.61:44427
I20260619 02:49:44.009792 1619 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.009910 1619 heartbeater.cc:511] Master 127.26.98.61:44427 requested a full tablet report, sending...
I20260619 02:49:44.010428 1249 ts_manager.cc:194] Registered new tserver with Master: 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
I20260619 02:49:44.011112 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.26.98.1:40509)
0619 02:49:43.991423 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.991449 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.992135 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.992512 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.992518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.992939 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.993279 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.993285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.994456 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.994457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995265 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.995268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995804 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.995904 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.996667 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.996678 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.007201 (+ 10523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.009745 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.009747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.009747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.009993 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.010275 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.010276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.010276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.010355 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.010688 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.010689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.010744 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.010802 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.010982 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":124,"thread_start_us":74,"threads_started":1}
I20260619 02:49:44.011111 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.991126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:40509 (local address 127.26.98.60:45421)
0619 02:49:43.991370 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.991372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.992320 (+ 948us) server_negotiation.cc:407] Connection header received
0619 02:49:43.992352 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.992353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.992368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.992438 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.993369 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.994385 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995343 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.995675 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.995741 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.007343 (+ 11602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.007346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.007346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.007359 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.009495 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.010112 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.010114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.010114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.010194 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.010422 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.010423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.010423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.010591 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.010659 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.010855 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.011007 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
I20260619 02:49:44.011642 1617 heartbeater.cc:347] Connected to a master server at 127.26.98.60:45421
I20260619 02:49:44.011696 1617 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.011754 1617 heartbeater.cc:511] Master 127.26.98.60:45421 requested a full tablet report, sending...
I20260619 02:49:44.012151 1438 ts_manager.cc:194] Registered new tserver with Master: 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
W20260619 02:49:44.108652 1625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:44.109014 1625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:44.109050 1625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:44.111656 1625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:44.111749 1625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:44.111809 1625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:44.114711 1625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:40415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=34129
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:44.115373 1625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:44.115726 1625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:44.122315 1633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:44.122361 1630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:44.122326 1631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:44.123170 1625 server_base.cc:1034] running on GCE node
I20260619 02:49:44.123495 1625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:44.123816 1625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:44.124997 1625 hybrid_clock.cc:648] HybridClock initialized: now 1781837384124983 us; error 39 us; skew 500 ppm
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:44.128409 1625 init.cc:377] Logged in from keytab as oryx/127.26.98.2@KRBTEST.COM (short username oryx)
I20260619 02:49:44.129086 1625 webserver.cc:460] Webserver started at http://127.26.98.2:34129/ using document root <none> and password file <none>
I20260619 02:49:44.129395 1625 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:44.129477 1625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:44.131379 1625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:49:44.132225 1640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.132575 1625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:44.132684 1625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "6ceec07f218e46b0bdad52c30c60430a"
format_stamp: "Formatted at 2026-06-19 02:49:42 on dist-test-slave-0f9d"
server_key: "ff45cad86b2601e963f6a5b91b6aa86c"
server_key_iv: "843bfed021335f08e086fc090dcb9dc3"
server_key_version: "encryptionkey@0"
I20260619 02:49:44.133194 1625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:44.142277 1625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:44.144882 1625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:44.145058 1625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:44.145344 1625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:44.145690 1625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:44.145725 1625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.145749 1625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:44.145772 1625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.153000 1625 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40415
I20260619 02:49:44.153045 1753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40415 every 8 connection(s)
I20260619 02:49:44.153487 1625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:44.155555 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 1625
I20260619 02:49:44.155788 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 985
I20260619 02:49:44.162520 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:37185
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:40659
--webserver_port=44999
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:44.169480 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:33095 (local address 127.26.98.61:44427)
0619 02:49:44.154558 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.154560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.154776 (+ 216us) server_negotiation.cc:407] Connection header received
0619 02:49:44.156704 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.156706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.156720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.156749 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.157663 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158480 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160489 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.160788 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160829 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.165445 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.165447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.165448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.165458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.167418 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168255 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.168256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.168257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.168321 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168580 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.168581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.168581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.168753 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.168815 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.169277 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.169380 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:49:44.169791 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.26.98.2:33095)
0619 02:49:44.154711 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.154720 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.156620 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.156928 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.156929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.157173 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.157544 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.157553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158617 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.158618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.160074 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.160077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.160916 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.161007 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.161773 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.161789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.165282 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.167566 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.167583 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.167588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.168153 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.168378 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.168379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.168510 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.168811 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.168820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.169072 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.169229 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.169535 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":190,"thread_start_us":123,"threads_started":1}
I20260619 02:49:44.170056 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:38753 (local address 127.26.98.60:45421)
0619 02:49:44.154456 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.154459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.154857 (+ 398us) server_negotiation.cc:407] Connection header received
0619 02:49:44.156455 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.156457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.156473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.156528 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.157755 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158711 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160209 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.160543 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160620 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.165445 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.165447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.165448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.165458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.167758 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168254 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.168256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.168256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.168337 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168618 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.168620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.168620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.168775 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.168837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.169846 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.169960 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:44.170185 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.26.98.2:38753)
0619 02:49:44.154599 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.154652 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.156207 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.156690 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.156697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.157353 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.157550 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.157557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158859 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.158861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.160080 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.160083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.160713 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.160875 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.161917 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.161923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.165282 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.167852 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.167853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.167853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.168142 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.168422 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.168423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.168424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.168527 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.168864 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.168865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.169079 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.169198 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.169466 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":7,"thread_start_us":109,"threads_started":1}
I20260619 02:49:44.170349 1756 heartbeater.cc:347] Connected to a master server at 127.26.98.61:44427
I20260619 02:49:44.170454 1756 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.170704 1756 heartbeater.cc:511] Master 127.26.98.61:44427 requested a full tablet report, sending...
I20260619 02:49:44.170818 1754 heartbeater.cc:347] Connected to a master server at 127.26.98.60:45421
I20260619 02:49:44.170900 1754 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.170962 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:52085 (local address 127.26.98.62:40659)
0619 02:49:44.155017 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.155019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.155267 (+ 248us) server_negotiation.cc:407] Connection header received
0619 02:49:44.156818 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.156820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.156834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.156864 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.158035 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158753 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160458 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.160776 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.160843 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.165440 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.165444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.165444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.165457 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.167779 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.169843 (+ 2064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.169845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.169846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.169918 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.170122 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.170123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.170124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.170270 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.170326 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.170742 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.170839 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:49:44.170975 1754 heartbeater.cc:511] Master 127.26.98.60:45421 requested a full tablet report, sending...
I20260619 02:49:44.171262 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.154782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:52085)
0619 02:49:44.155091 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.155102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.156570 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.157659 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.157660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.157854 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.157933 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.157935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.158839 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.158840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.160398 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.160399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.161244 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.161244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.161317 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.161850 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.161854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.165228 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.169522 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.169523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.169524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.169735 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.169985 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.169986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.169986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.170041 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.170571 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.170572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.170622 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.171037 (+ 415us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.171170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":494,"thread_start_us":76,"threads_started":1}
I20260619 02:49:44.171324 1249 ts_manager.cc:194] Registered new tserver with Master: 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
I20260619 02:49:44.171351 1438 ts_manager.cc:194] Registered new tserver with Master: 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
I20260619 02:49:44.171650 1755 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:44.171694 1755 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.171759 1755 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:44.171999 654 ts_manager.cc:194] Re-registered known tserver with Master: 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
I20260619 02:49:44.172415 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:52085
I20260619 02:49:44.267663 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.263256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.26.98.60:45421)
0619 02:49:44.263322 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.263324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.263509 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:49:44.263578 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.263579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.263613 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.263713 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.263892 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.264745 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.266275 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.267027 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.267128 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.267234 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.267487 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.267552 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:44.267663 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.263324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:58180)
0619 02:49:44.263408 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.263422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.263513 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.263724 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.263725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.263727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.263817 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.263820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.264882 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.264884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.266045 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.266051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.267192 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.267195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.267366 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.267380 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.267466 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.267556 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:49:44.268312 1448 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 02:49:44.268581 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 392149059ead4374b22e3c3bab158a95. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:44.268707 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: LEARNER
W20260619 02:49:44.273365 1761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:44.273666 1761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:44.273711 1761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:44.275606 1761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:44.275681 1761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:44.275735 1761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:44.277971 1761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:37185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=44999
--tserver_master_addrs=127.26.98.62:40659,127.26.98.61:44427,127.26.98.60:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:44.278314 1761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:44.278582 1761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:44.285336 1769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:44.285321 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:44.285321 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:44.285938 1761 server_base.cc:1034] running on GCE node
I20260619 02:49:44.286284 1761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:44.286638 1761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:44.287848 1761 hybrid_clock.cc:648] HybridClock initialized: now 1781837384287829 us; error 41 us; skew 500 ppm
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:44.291460 1761 init.cc:377] Logged in from keytab as oryx/127.26.98.3@KRBTEST.COM (short username oryx)
I20260619 02:49:44.292107 1761 webserver.cc:460] Webserver started at http://127.26.98.3:44999/ using document root <none> and password file <none>
I20260619 02:49:44.292480 1761 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:44.292893 1761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:44.295185 1761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:44.296468 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.297569 1761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:44.297741 1761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "62d104c829684c7793489e48278a1018"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-0f9d"
server_key: "d53a951db07243ce563a8ad47246597f"
server_key_iv: "ca06de13b2a72cd591723fe9a7e73f83"
server_key_version: "encryptionkey@0"
I20260619 02:49:44.299134 1761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:44.313364 1761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:44.316574 1761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:44.316757 1761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:44.317044 1761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:44.317442 1761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:44.317480 1761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.317517 1761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:44.317543 1761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.325544 1761 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:37185
I20260619 02:49:44.325632 1889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:37185 every 8 connection(s)
I20260619 02:49:44.326064 1761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:44.333261 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 1761
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.60@KRBTEST.COM
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.61@KRBTEST.COM
Jun 19 02:49:44 dist-test-slave-0f9d krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837384, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:44.341269 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.327030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:52863)
0619 02:49:44.327485 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.327543 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.328982 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.329387 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.329394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.329879 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.330197 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.330205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.331250 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.331250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.332367 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.332369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.332754 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.332920 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.334809 (+ 1889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.334815 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.336988 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.339075 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.339094 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.339098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.339550 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.339833 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.339834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.339835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.340015 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.340347 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.340354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.340594 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.340748 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.340964 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":67,"thread_start_us":136,"threads_started":1}
I20260619 02:49:44.342080 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.327341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:52863 (local address 127.26.98.62:40659)
0619 02:49:44.327398 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.327400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.327697 (+ 297us) server_negotiation.cc:407] Connection header received
0619 02:49:44.329155 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.329157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.329172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.329258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.330349 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.331118 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332453 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.332671 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332735 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.337103 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.337106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.337106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.337117 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.338954 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.339677 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.339680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.339681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.339755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.340089 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.340090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.340090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.340245 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.340317 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.341862 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.341982 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:44.342217 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.330279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.26.98.3:36725)
0619 02:49:44.330561 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.330573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.330653 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.330819 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.330820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.331069 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.331149 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.331151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.332110 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.332960 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.332962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.333475 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.333476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.333554 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.334031 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.334037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.337224 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.340955 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.340956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.340956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.341184 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.341400 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.341401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.341465 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.341820 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.341821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.341879 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.341946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.342083 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":140,"threads_started":1}
I20260619 02:49:44.342453 1891 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40659
I20260619 02:49:44.342541 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.330434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:36725 (local address 127.26.98.61:44427)
0619 02:49:44.330492 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.330494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.330662 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:49:44.330705 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.330706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.330721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.330795 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.331239 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.331996 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.333044 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.333327 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.333403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.337448 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.337450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.337450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.337460 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.340760 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341267 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.341268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.341268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.341331 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341548 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.341549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.341550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.341716 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.341789 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.342359 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.342452 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:44.342561 1891 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.342682 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:51145 (local address 127.26.98.60:45421)
0619 02:49:44.327398 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.327400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.329331 (+ 1931us) server_negotiation.cc:407] Connection header received
0619 02:49:44.329384 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.329385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.329400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.329460 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.330338 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.331151 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332474 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.332827 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332871 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.337273 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.337275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.337275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.337284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.340008 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341827 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.341829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.341829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.341898 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.342088 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.342089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.342090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.342241 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.342289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.342495 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.342594 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:44.342824 1891 heartbeater.cc:511] Master 127.26.98.62:40659 requested a full tablet report, sending...
I20260619 02:49:44.343093 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.327030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.26.98.3:51145)
0619 02:49:44.329260 (+ 2230us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.329273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.329335 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.329462 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.329462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.329886 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.330220 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.330223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.331257 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.331258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.332367 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.332369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.332955 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.332956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.333014 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.333650 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.333662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.337187 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.341542 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.341543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.341735 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.341970 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.341971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.341972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.342026 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.342337 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.342338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.342383 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.342807 (+ 424us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.342991 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2190,"mutex_wait_us":6,"thread_start_us":109,"threads_started":1}
I20260619 02:49:44.343214 1892 heartbeater.cc:347] Connected to a master server at 127.26.98.61:44427
I20260619 02:49:44.343250 1892 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.343322 1892 heartbeater.cc:511] Master 127.26.98.61:44427 requested a full tablet report, sending...
I20260619 02:49:44.343400 654 ts_manager.cc:194] Re-registered known tserver with Master: 62d104c829684c7793489e48278a1018 (127.26.98.3:37185)
I20260619 02:49:44.343565 1890 heartbeater.cc:347] Connected to a master server at 127.26.98.60:45421
I20260619 02:49:44.343614 1890 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:44.343677 1890 heartbeater.cc:511] Master 127.26.98.60:45421 requested a full tablet report, sending...
I20260619 02:49:44.343928 654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:52863
I20260619 02:49:44.343992 1438 ts_manager.cc:194] Registered new tserver with Master: 62d104c829684c7793489e48278a1018 (127.26.98.3:37185)
I20260619 02:49:44.344930 1249 ts_manager.cc:194] Registered new tserver with Master: 62d104c829684c7793489e48278a1018 (127.26.98.3:37185)
I20260619 02:49:44.347735 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.335169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.26.98.62:40659)
0619 02:49:44.335668 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.335671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.335801 (+ 130us) server_negotiation.cc:407] Connection header received
0619 02:49:44.335852 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.335853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.335867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.336031 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.338127 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.339271 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.340696 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.341038 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.341114 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.343776 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.343778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.343778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.343788 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.346400 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.346822 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.346823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.346824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.346920 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.347134 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.347136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.347136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.347293 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.347404 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.347505 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.347618 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":111,"threads_started":1}
I20260619 02:49:44.349720 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.336517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.26.98.61:44427)
0619 02:49:44.336674 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.336677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.336687 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:44.336721 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.336722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.336740 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.336831 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.338296 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.339301 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.340698 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.341038 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.346296 (+ 5258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.346362 (+ 66us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.346363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.346363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.346369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.348140 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.348519 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.348520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.348522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.348596 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.349075 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.349076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.349076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.349251 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.349397 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.349499 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.349610 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260619 02:49:44.350219 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.334578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.26.98.60:45421)
0619 02:49:44.334815 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.334818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.335731 (+ 913us) server_negotiation.cc:407] Connection header received
0619 02:49:44.335845 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.335846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.335863 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.335927 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.338098 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.344466 (+ 6368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.345549 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.345779 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.345839 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.347211 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.347213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.347214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.347220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.348933 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.349307 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.349308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.349309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.349375 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.349559 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.349560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.349560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.349722 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.349821 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.349921 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.350015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260619 02:49:44.349803 654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:44.352447 654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:44.353513 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: NON_VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: true } } }
I20260619 02:49:44.353652 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:44.355160 1909 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: attempting to promote NON_VOTER 28b079c8bda844f79b2ac1c702946dc3 to VOTER
I20260619 02:49:44.355332 1909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } }
I20260619 02:49:44.355783 1448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 LEARNER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:49:44.355784 1262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Refusing update from remote peer 392149059ead4374b22e3c3bab158a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 02:49:44.356009 1404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 02:49:44.356236 1909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 02:49:44.357064 1909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.357198 1262 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.357587 1448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.358328 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.358410 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:44.358491 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.358542 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:44.358520 1914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.358592 1914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:44.358759 1918 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.358848 1918 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:49:44.359030 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392149059ead4374b22e3c3bab158a95" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:44.359123 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:49:44.364149 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36779 (local address 127.0.0.1:40344)
0619 02:49:44.359597 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.359616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.359671 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.360464 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.360464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.360467 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.360526 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.360528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.361545 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.361547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.362858 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.362862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.363805 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.363806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.363876 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.363884 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.363948 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.364007 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1,"thread_start_us":143,"threads_started":1}
I20260619 02:49:44.364137 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.26.98.1:36779)
0619 02:49:44.360283 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.360288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.360299 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:44.360342 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.360345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.360361 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.360587 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.360648 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.361425 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.362980 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.363686 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.363778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.363862 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.363968 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.364029 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":110,"threads_started":1}
I20260619 02:49:44.365134 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37185 (local address 127.0.0.1:53196)
0619 02:49:44.359225 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.359239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.359280 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.360148 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.360149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.360152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.360247 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.360250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.361729 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.361730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.362858 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.362862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.364871 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.364872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.364946 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.364952 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.364981 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.365039 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":25}
I20260619 02:49:44.365279 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40415 (local address 127.0.0.1:52188)
0619 02:49:44.359784 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.359794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.359847 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.360360 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.360361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.360363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.360408 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.360409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.362996 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.362997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.363780 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.363784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.364948 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.364950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.365029 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.365035 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.365096 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.365151 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260619 02:49:44.365309 1551 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3d22e3243f43f58603734438f24696 (DEFAULT_TABLE table=test-table [id=641c7127644a4ae7b73dd8fc205dbcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.365615 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.26.98.2:40415)
0619 02:49:44.360127 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.360132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.360142 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:44.360182 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.360184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.360200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.360289 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.360484 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.361638 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.363905 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.364851 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.364928 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.365031 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.365116 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.365190 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260619 02:49:44.365689 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.359326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.26.98.3:37185)
0619 02:49:44.359865 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.359879 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.359895 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:49:44.359955 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.359958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.359980 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.360326 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.360398 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.361637 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.363912 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.364756 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.365384 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.365494 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.365527 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.365588 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":151,"threads_started":1}
I20260619 02:49:44.365872 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3d22e3243f43f58603734438f24696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.366300 1688 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3d22e3243f43f58603734438f24696 (DEFAULT_TABLE table=test-table [id=641c7127644a4ae7b73dd8fc205dbcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.366647 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3d22e3243f43f58603734438f24696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.366560 1824 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3d22e3243f43f58603734438f24696 (DEFAULT_TABLE table=test-table [id=641c7127644a4ae7b73dd8fc205dbcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.366986 1824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3d22e3243f43f58603734438f24696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.370608 1924 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Bootstrap starting.
I20260619 02:49:44.371268 1925 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Bootstrap starting.
I20260619 02:49:44.371425 1926 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Bootstrap starting.
I20260619 02:49:44.371704 1924 tablet_bootstrap.cc:654] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.372051 1924 log.cc:824] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:44.372160 1926 tablet_bootstrap.cc:654] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.372339 1925 tablet_bootstrap.cc:654] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.372916 1926 log.cc:824] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:44.372984 1924 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: No bootstrap required, opened a new log
I20260619 02:49:44.372994 1925 log.cc:824] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:44.373068 1924 ts_tablet_manager.cc:1397] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:44.373984 1926 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: No bootstrap required, opened a new log
I20260619 02:49:44.374082 1926 ts_tablet_manager.cc:1397] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:44.374325 1925 tablet_bootstrap.cc:492] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: No bootstrap required, opened a new log
I20260619 02:49:44.374405 1925 ts_tablet_manager.cc:1397] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:44.375849 1924 raft_consensus.cc:348] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.376040 1924 raft_consensus.cc:374] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.376073 1924 raft_consensus.cc:729] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ceec07f218e46b0bdad52c30c60430a, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.376214 1924 consensus_queue.cc:260] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.376567 1924 ts_tablet_manager.cc:1428] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:44.376686 1755 heartbeater.cc:503] Master 127.26.98.62:40659 was elected leader, sending a full tablet report...
I20260619 02:49:44.377076 1925 raft_consensus.cc:348] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.377147 1926 raft_consensus.cc:348] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.377282 1926 raft_consensus.cc:374] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.377283 1925 raft_consensus.cc:374] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.377315 1925 raft_consensus.cc:729] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3777adda73a14d3f8f79616234029d65, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.377315 1926 raft_consensus.cc:729] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d104c829684c7793489e48278a1018, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.377461 1925 consensus_queue.cc:260] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.377461 1926 consensus_queue.cc:260] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.377765 1925 ts_tablet_manager.cc:1428] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 02:49:44.377805 1926 ts_tablet_manager.cc:1428] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 02:49:44.378108 1618 heartbeater.cc:503] Master 127.26.98.62:40659 was elected leader, sending a full tablet report...
I20260619 02:49:44.378659 1891 heartbeater.cc:503] Master 127.26.98.62:40659 was elected leader, sending a full tablet report...
I20260619 02:49:44.385866 1932 raft_consensus.cc:482] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:44.386022 1932 raft_consensus.cc:504] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.386458 1932 leader_election.cc:288] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779), 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
I20260619 02:49:44.390640 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.386769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:44919 (local address 127.26.98.1:36779)
0619 02:49:44.386820 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.386822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.386834 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:44.386935 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.386936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.386952 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.387017 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.387474 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.388229 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.389565 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.390205 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.390277 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.390293 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.390517 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.390571 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:44.390656 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.386687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36779 (local address 127.26.98.3:44919)
0619 02:49:44.386770 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.386784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.386838 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.387043 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.387044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.387046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.387237 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.387244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.388355 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.388356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.389430 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.389433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.390310 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.390311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.390410 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.390416 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.390493 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.390553 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:49:44.391224 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.386824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40415 (local address 127.26.98.3:36807)
0619 02:49:44.386866 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.386904 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.386964 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.387508 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.387510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.387512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.387958 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.387961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.388968 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.388969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.390049 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.390052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.390913 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.390914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.391009 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.391013 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.391081 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.391130 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 02:49:44.391233 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.386978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:36807 (local address 127.26.98.2:40415)
0619 02:49:44.387035 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.387037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.387045 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:44.387217 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.387220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.387239 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.387359 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.388118 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.388793 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.390178 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.390803 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.390915 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.390930 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.391101 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.391162 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:44.391549 1708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3d22e3243f43f58603734438f24696" candidate_uuid: "62d104c829684c7793489e48278a1018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ceec07f218e46b0bdad52c30c60430a" is_pre_election: true
I20260619 02:49:44.391527 1571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3d22e3243f43f58603734438f24696" candidate_uuid: "62d104c829684c7793489e48278a1018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3777adda73a14d3f8f79616234029d65" is_pre_election: true
I20260619 02:49:44.391749 1571 raft_consensus.cc:2454] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62d104c829684c7793489e48278a1018 in term 0.
I20260619 02:49:44.391749 1708 raft_consensus.cc:2454] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62d104c829684c7793489e48278a1018 in term 0.
I20260619 02:49:44.392096 1778 leader_election.cc:302] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3777adda73a14d3f8f79616234029d65, 62d104c829684c7793489e48278a1018; no voters:
I20260619 02:49:44.392303 1932 raft_consensus.cc:2781] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:44.392370 1932 raft_consensus.cc:482] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:44.392388 1932 raft_consensus.cc:3037] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.393256 1932 raft_consensus.cc:504] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.393405 1932 leader_election.cc:288] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [CANDIDATE]: Term 1 election: Requested vote from peers 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779), 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415)
I20260619 02:49:44.393661 1571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3d22e3243f43f58603734438f24696" candidate_uuid: "62d104c829684c7793489e48278a1018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3777adda73a14d3f8f79616234029d65"
I20260619 02:49:44.393663 1708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3d22e3243f43f58603734438f24696" candidate_uuid: "62d104c829684c7793489e48278a1018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ceec07f218e46b0bdad52c30c60430a"
I20260619 02:49:44.393747 1571 raft_consensus.cc:3037] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.393777 1708 raft_consensus.cc:3037] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.394651 1708 raft_consensus.cc:2454] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62d104c829684c7793489e48278a1018 in term 1.
I20260619 02:49:44.394656 1571 raft_consensus.cc:2454] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62d104c829684c7793489e48278a1018 in term 1.
I20260619 02:49:44.394860 1778 leader_election.cc:302] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3777adda73a14d3f8f79616234029d65, 62d104c829684c7793489e48278a1018; no voters:
I20260619 02:49:44.395011 1932 raft_consensus.cc:2781] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:44.395218 1932 raft_consensus.cc:686] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 1 LEADER]: Becoming Leader. State: Replica: 62d104c829684c7793489e48278a1018, State: Running, Role: LEADER
I20260619 02:49:44.395336 1932 consensus_queue.cc:237] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } }
I20260619 02:49:44.396354 655 catalog_manager.cc:5495] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62d104c829684c7793489e48278a1018 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "62d104c829684c7793489e48278a1018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:44.404397 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.401739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.26.98.3:37185)
0619 02:49:44.401789 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.401791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.401801 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:44.401852 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.401853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.401878 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.401944 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:44.402157 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.402723 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.403554 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.403814 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.403871 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.404032 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.404178 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.404259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.404275 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.404301 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 02:49:44.404397 1757 tablet.cc:2369] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:44.406523 1708 raft_consensus.cc:1261] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Refusing update from remote peer 62d104c829684c7793489e48278a1018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.406523 1571 raft_consensus.cc:1261] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Refusing update from remote peer 62d104c829684c7793489e48278a1018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.407034 1933 consensus_queue.cc:1035] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.407263 1932 consensus_queue.cc:1035] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.411110 1936 mvcc.cc:205] Tried to move back new op lower bound from 7298405926526164992 to 7298405926484176896. Current Snapshot: MvccSnapshot[applied={T|T < 7298405926526164992}]
I20260619 02:49:44.411152 1937 mvcc.cc:205] Tried to move back new op lower bound from 7298405926526164992 to 7298405926484176896. Current Snapshot: MvccSnapshot[applied={T|T < 7298405926526164992}]
I20260619 02:49:44.412510 1938 mvcc.cc:205] Tried to move back new op lower bound from 7298405926526164992 to 7298405926484176896. Current Snapshot: MvccSnapshot[applied={T|T < 7298405926526164992}]
I20260619 02:49:44.416051 654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:44.416257 654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:44.418107 654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 392149059ead4374b22e3c3bab158a95: Sending DeleteTablet for 3 replicas of tablet 1a3d22e3243f43f58603734438f24696
I20260619 02:49:44.418668 1824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3d22e3243f43f58603734438f24696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:53196
I20260619 02:49:44.418664 1688 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3d22e3243f43f58603734438f24696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52188
I20260619 02:49:44.418700 1551 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3d22e3243f43f58603734438f24696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:40344
I20260619 02:49:44.419040 1948 tablet_replica.cc:330] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: stopping tablet replica
I20260619 02:49:44.419344 1948 raft_consensus.cc:2229] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:44.419508 1948 raft_consensus.cc:2258] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.419890 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 623
I20260619 02:49:44.420811 1948 ts_tablet_manager.cc:1905] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.422832 1948 ts_tablet_manager.cc:1918] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:44.422982 1948 log.cc:1196] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/1a3d22e3243f43f58603734438f24696
I20260619 02:49:44.423271 1948 ts_tablet_manager.cc:1939] T 1a3d22e3243f43f58603734438f24696 P 3777adda73a14d3f8f79616234029d65: Deleting consensus metadata
W20260619 02:49:44.426101 1505 connection.cc:535] server connection from 127.0.0.1:40344 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:49:44.431047 1946 tablet_replica.cc:330] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: stopping tablet replica
I20260619 02:49:44.431263 1946 raft_consensus.cc:2229] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:44.431500 1946 raft_consensus.cc:2258] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.432097 1947 tablet_replica.cc:330] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: stopping tablet replica
I20260619 02:49:44.432163 1946 ts_tablet_manager.cc:1905] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.432272 1947 raft_consensus.cc:2229] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:44.432417 1947 raft_consensus.cc:2258] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.434558 1946 ts_tablet_manager.cc:1918] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:44.434659 1946 log.cc:1196] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/1a3d22e3243f43f58603734438f24696
I20260619 02:49:44.435020 1946 ts_tablet_manager.cc:1939] T 1a3d22e3243f43f58603734438f24696 P 62d104c829684c7793489e48278a1018: Deleting consensus metadata
I20260619 02:49:44.435935 1947 ts_tablet_manager.cc:1905] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.436049 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.427543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.26.98.60:45421)
0619 02:49:44.427602 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.427604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.427901 (+ 297us) server_negotiation.cc:407] Connection header received
0619 02:49:44.427976 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.427977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.427989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.428039 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.428727 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.429269 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.430323 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.430595 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.430661 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.432921 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.432923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.432923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.432931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.434704 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.435284 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.435285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.435286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.435347 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.435532 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.435533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.435533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.435656 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.435757 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.435846 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.435926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:49:44.438059 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.427778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.26.98.61:44427)
0619 02:49:44.427850 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.427852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.428083 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:49:44.428193 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.428195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.428211 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.428260 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.429404 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.430181 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.431266 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.431567 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.431620 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.433841 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.433843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.433844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.433854 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.435792 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.437155 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.437156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.437157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.437243 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.437467 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.437468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.437469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.437665 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.437771 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.437862 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.437952 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:44.438048 1947 ts_tablet_manager.cc:1918] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:44.438133 1947 log.cc:1196] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/1a3d22e3243f43f58603734438f24696
I20260619 02:49:44.438474 1947 ts_tablet_manager.cc:1939] T 1a3d22e3243f43f58603734438f24696 P 6ceec07f218e46b0bdad52c30c60430a: Deleting consensus metadata
W20260619 02:49:44.438959 1642 connection.cc:441] server connection from 127.0.0.1:52188 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52188 (request call id 1) could send its response
W20260619 02:49:44.439081 1780 connection.cc:441] server connection from 127.0.0.1:53196 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53196 (request call id 1) could send its response
I20260619 02:49:44.439810 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.419235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.26.98.60:45421)
0619 02:49:44.419286 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.419288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.431375 (+ 12087us) server_negotiation.cc:407] Connection header received
0619 02:49:44.431468 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.431469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.431487 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.431518 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.433041 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.433674 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.434696 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.435013 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.435059 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.436760 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.436761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.436762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.436768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.438414 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.438855 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.438856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.438856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.438955 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.439306 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.439307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.439307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.439442 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.439537 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.439661 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.439737 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:45.413065 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.412650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:52569)
0619 02:49:45.412865 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.412985 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260619 02:49:45.413398 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.413008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:45571)
0619 02:49:45.413115 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.413343 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
W20260619 02:49:45.413398 1755 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:40659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.413452 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.413184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:34223)
0619 02:49:45.413311 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.413400 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
W20260619 02:49:45.413673 1618 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:40659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
W20260619 02:49:45.413805 1891 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:40659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.918749 1960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 392149059ead4374b22e3c3bab158a95)
I20260619 02:49:45.918988 1960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } }
I20260619 02:49:45.919231 1961 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 392149059ead4374b22e3c3bab158a95)
I20260619 02:49:45.919318 1960 leader_election.cc:288] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659), de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61:44427)
I20260619 02:49:45.919394 1961 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } }
I20260619 02:49:45.919909 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.919514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40862)
0619 02:49:45.919620 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.919852 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260619 02:49:45.920080 1961 leader_election.cc:288] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659), 28b079c8bda844f79b2ac1c702946dc3 (127.26.98.60:45421)
W20260619 02:49:45.920154 1424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.920292 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.919815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40872)
0619 02:49:45.920206 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.920251 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":64,"threads_started":1}
W20260619 02:49:45.920413 1238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.920646 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.920541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40884)
0619 02:49:45.920583 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.920616 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31,"thread_start_us":85,"threads_started":1}
W20260619 02:49:45.920802 1424 leader_election.cc:334] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659): Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.920907 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.920826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40888)
0619 02:49:45.920862 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.920883 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:49:45.920984 1238 leader_election.cc:334] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659): Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.923400 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.919707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44427 (local address 127.0.0.1:37002)
0619 02:49:45.919839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.919850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:45.919918 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:45.920152 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:45.920154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:45.920157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:45.920248 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.920251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.921183 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.921184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.922418 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.922420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.923163 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.923165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.923244 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.923249 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:45.923295 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:45.923338 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
I20260619 02:49:45.923439 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.919912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.26.98.61:44427)
0619 02:49:45.919991 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.919993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.920003 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:45.920032 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.920033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.920044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.920143 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:45.920329 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.921104 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.922531 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.923068 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.923166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.923183 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.923315 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.923363 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"thread_start_us":29,"threads_started":1}
I20260619 02:49:45.923667 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28b079c8bda844f79b2ac1c702946dc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de9778a2c2894229b7e747b8a9b7f42c" is_pre_election: true
I20260619 02:49:45.923808 1262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b079c8bda844f79b2ac1c702946dc3 in term 1.
I20260619 02:49:45.923857 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.920002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.26.98.60:45421)
0619 02:49:45.920180 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.920182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.920189 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:45.920205 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.920206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.920214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.920290 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:45.920896 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.921594 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.922914 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.923459 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.923552 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.923567 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.923743 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.923791 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":55,"threads_started":1}
I20260619 02:49:45.923885 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.919804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:45421 (local address 127.0.0.1:48268)
0619 02:49:45.920011 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.920021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:45.920072 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:45.920736 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:45.920738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:45.920740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:45.920798 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.920801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.921698 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.921699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.922786 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.922788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.923599 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.923600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.923670 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.923674 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:45.923730 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:45.923780 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":49,"threads_started":1}
I20260619 02:49:45.924023 1424 leader_election.cc:302] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b079c8bda844f79b2ac1c702946dc3, de9778a2c2894229b7e747b8a9b7f42c; no voters: 392149059ead4374b22e3c3bab158a95
I20260619 02:49:45.924099 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de9778a2c2894229b7e747b8a9b7f42c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28b079c8bda844f79b2ac1c702946dc3" is_pre_election: true
I20260619 02:49:45.924160 1960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:49:45.924198 1960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 392149059ead4374b22e3c3bab158a95)
I20260619 02:49:45.924216 1960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:49:45.925076 1960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } }
I20260619 02:49:45.925222 1960 leader_election.cc:288] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 election: Requested vote from peers 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659), de9778a2c2894229b7e747b8a9b7f42c (127.26.98.61:44427)
I20260619 02:49:45.925244 1448 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de9778a2c2894229b7e747b8a9b7f42c in current term 2: Already voted for candidate 28b079c8bda844f79b2ac1c702946dc3 in this term.
I20260619 02:49:45.925488 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.925384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40890)
0619 02:49:45.925430 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.925462 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:49:45.925570 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28b079c8bda844f79b2ac1c702946dc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de9778a2c2894229b7e747b8a9b7f42c"
I20260619 02:49:45.925650 1262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:49:45.925791 1236 leader_election.cc:302] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de9778a2c2894229b7e747b8a9b7f42c; no voters: 28b079c8bda844f79b2ac1c702946dc3, 392149059ead4374b22e3c3bab158a95
I20260619 02:49:45.926354 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.926245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40894)
0619 02:49:45.926297 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.926329 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 02:49:45.926443 1262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b079c8bda844f79b2ac1c702946dc3 in term 2.
W20260619 02:49:45.926452 1424 leader_election.cc:334] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659): Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
I20260619 02:49:45.926529 1961 raft_consensus.cc:2726] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:49:45.926585 1424 leader_election.cc:302] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b079c8bda844f79b2ac1c702946dc3, de9778a2c2894229b7e747b8a9b7f42c; no voters: 392149059ead4374b22e3c3bab158a95
I20260619 02:49:45.926651 1960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:49:45.926826 1960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [term 2 LEADER]: Becoming Leader. State: Replica: 28b079c8bda844f79b2ac1c702946dc3, State: Running, Role: LEADER
I20260619 02:49:45.926919 1960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } }
I20260619 02:49:45.927280 1971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b079c8bda844f79b2ac1c702946dc3. Latest consensus state: current_term: 2 leader_uuid: "28b079c8bda844f79b2ac1c702946dc3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:45.927389 1971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:45.927574 1973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:45.928042 1973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:45.928110 1973 catalog_manager.cc:1259] Loaded cluster ID: f10cdd7eb55a4edfa2833f5025c627cf
I20260619 02:49:45.928141 1973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:45.928325 1973 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:45.928406 1973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Loaded TSK: 0
I20260619 02:49:45.928547 1973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:45.940997 1437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:45.943217 1437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:45.944730 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.944586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40912)
0619 02:49:45.944640 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.944688 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 02:49:45.944698 1262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [term 2 FOLLOWER]: Refusing update from remote peer 28b079c8bda844f79b2ac1c702946dc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 02:49:45.945062 1971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 02:49:45.945240 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.945111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.0.0.1:40914)
0619 02:49:45.945167 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.945212 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:49:45.945468 1424 consensus_peers.cc:487] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 -> Peer 392149059ead4374b22e3c3bab158a95 (127.26.98.62:40659): Couldn't send request to peer 392149059ead4374b22e3c3bab158a95. Status: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:49:45.946017 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b079c8bda844f79b2ac1c702946dc3. Latest consensus state: current_term: 2 leader_uuid: "28b079c8bda844f79b2ac1c702946dc3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:45.946127 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:49:45.946874 1974 mvcc.cc:205] Tried to move back new op lower bound from 7298405932826779648 to 7298405932757311488. Current Snapshot: MvccSnapshot[applied={T|T < 7298405932826779648}]
I20260619 02:49:45.947065 1960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "28b079c8bda844f79b2ac1c702946dc3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:45.947162 1960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:45.947206 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "28b079c8bda844f79b2ac1c702946dc3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "392149059ead4374b22e3c3bab158a95" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40659 } } peers { permanent_uuid: "de9778a2c2894229b7e747b8a9b7f42c" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44427 } attrs { promote: false } } peers { permanent_uuid: "28b079c8bda844f79b2ac1c702946dc3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 45421 } attrs { promote: false } } }
I20260619 02:49:45.947321 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9778a2c2894229b7e747b8a9b7f42c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:49:45.947448 1975 mvcc.cc:205] Tried to move back new op lower bound from 7298405932826779648 to 7298405932757311488. Current Snapshot: MvccSnapshot[applied={T|T < 7298405932826779648 or (T in {7298405932826779648})}]
I20260619 02:49:45.954079 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.950327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.26.98.1:36779)
0619 02:49:45.950484 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.950486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.950596 (+ 110us) server_negotiation.cc:407] Connection header received
0619 02:49:45.950627 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.950628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.950639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.951176 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:45.951368 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.951912 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.953021 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.953612 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.953685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.953701 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.953952 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.954003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260619 02:49:45.954093 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.950253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36779 (local address 127.0.0.1:55426)
0619 02:49:45.950476 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.950486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:45.950541 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:45.951174 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:45.951175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:45.951177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:45.951274 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.951276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.951989 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.951990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.952805 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.952807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.953695 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.953696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.953854 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.953862 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:49:45.953935 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:49:45.953989 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 02:49:45.954392 1551 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37c4db4a494390a7a9088906a150ce (DEFAULT_TABLE table=test-table [id=145b33b0435347349324d453a43605b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:45.954455 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.949991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37185 (local address 127.0.0.1:37236)
0619 02:49:45.950141 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.950203 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:45.950270 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:45.951426 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:45.951427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:45.951428 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:45.951467 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.951468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.952513 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.952515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.953425 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.953427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.954241 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.954242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.954305 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.954310 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:45.954341 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 02:49:45.954392 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 02:49:45.954528 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.950000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40415 (local address 127.0.0.1:57076)
0619 02:49:45.950319 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.950329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:45.950386 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:45.950638 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:45.950638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:45.950641 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:45.950728 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.950730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.951543 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.951544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.952853 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:45.952856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.954277 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.954278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:45.954342 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.954345 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:45.954385 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 02:49:45.954432 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":77}
I20260619 02:49:45.954555 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37c4db4a494390a7a9088906a150ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:45.954743 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.950362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.26.98.3:37185)
0619 02:49:45.951296 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.951298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.951307 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:45.951334 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.951335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.951348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.951416 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:45.951520 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.952399 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.953512 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.954120 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.954585 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.954606 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.954624 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.954675 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":82,"threads_started":1}
I20260619 02:49:45.954789 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.950112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.26.98.2:40415)
0619 02:49:45.950483 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.950485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.950493 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:45.950529 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.950530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.950542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.950631 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:45.950797 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.951435 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.952982 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:45.953593 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:45.953687 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:45.953702 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.954638 (+ 936us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.954712 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":111,"threads_started":1}
I20260619 02:49:45.955001 1824 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37c4db4a494390a7a9088906a150ce (DEFAULT_TABLE table=test-table [id=145b33b0435347349324d453a43605b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:45.955029 1688 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37c4db4a494390a7a9088906a150ce (DEFAULT_TABLE table=test-table [id=145b33b0435347349324d453a43605b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:45.955121 1824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37c4db4a494390a7a9088906a150ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:45.955170 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37c4db4a494390a7a9088906a150ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:45.956336 1982 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: Bootstrap starting.
I20260619 02:49:45.956799 1982 tablet_bootstrap.cc:654] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:45.957119 1983 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Bootstrap starting.
I20260619 02:49:45.957440 1982 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: No bootstrap required, opened a new log
I20260619 02:49:45.957487 1982 ts_tablet_manager.cc:1397] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:45.957507 1983 tablet_bootstrap.cc:654] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:45.957665 1982 raft_consensus.cc:348] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.957741 1982 raft_consensus.cc:374] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:45.957760 1982 raft_consensus.cc:729] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3777adda73a14d3f8f79616234029d65, State: Initialized, Role: FOLLOWER
I20260619 02:49:45.957801 1984 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Bootstrap starting.
I20260619 02:49:45.957847 1982 consensus_queue.cc:260] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.958026 1982 ts_tablet_manager.cc:1428] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:45.958266 1984 tablet_bootstrap.cc:654] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:45.958494 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.958254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:49097)
0619 02:49:45.958404 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.958451 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260619 02:49:45.958670 1983 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: No bootstrap required, opened a new log
I20260619 02:49:45.958719 1983 ts_tablet_manager.cc:1397] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:45.958866 1617 heartbeater.cc:503] Master 127.26.98.60:45421 was elected leader, sending a full tablet report...
I20260619 02:49:45.958947 1983 raft_consensus.cc:348] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.959020 1983 raft_consensus.cc:374] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:45.959039 1983 raft_consensus.cc:729] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ceec07f218e46b0bdad52c30c60430a, State: Initialized, Role: FOLLOWER
I20260619 02:49:45.959100 1983 consensus_queue.cc:260] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.959311 1983 ts_tablet_manager.cc:1428] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:45.959062 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.958975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:54845)
0619 02:49:45.959008 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.959036 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:49:45.959882 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.959654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:41717)
0619 02:49:45.959803 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.959841 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260619 02:49:45.960075 1754 heartbeater.cc:503] Master 127.26.98.60:45421 was elected leader, sending a full tablet report...
I20260619 02:49:45.960289 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.960197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:41265)
0619 02:49:45.960232 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.960263 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 02:49:45.960374 1984 tablet_bootstrap.cc:492] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: No bootstrap required, opened a new log
I20260619 02:49:45.960450 1984 ts_tablet_manager.cc:1397] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:45.960583 1984 raft_consensus.cc:348] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.960646 1984 raft_consensus.cc:374] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:45.960664 1984 raft_consensus.cc:729] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d104c829684c7793489e48278a1018, State: Initialized, Role: FOLLOWER
I20260619 02:49:45.960712 1984 consensus_queue.cc:260] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:45.960858 1984 ts_tablet_manager.cc:1428] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:45.961391 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.961120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:36867)
0619 02:49:45.961249 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.961289 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":88,"threads_started":1}
I20260619 02:49:45.961486 1890 heartbeater.cc:503] Master 127.26.98.60:45421 was elected leader, sending a full tablet report...
I20260619 02:49:45.961799 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.961684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:38523)
0619 02:49:45.961735 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:45.961771 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:49:45.992401 1620 tablet.cc:2369] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:46.048069 1990 raft_consensus.cc:482] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:46.048239 1990 raft_consensus.cc:504] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:46.048830 1990 leader_election.cc:288] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62d104c829684c7793489e48278a1018 (127.26.98.3:37185), 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
I20260619 02:49:46.053354 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.049019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36779 (local address 127.26.98.2:60175)
0619 02:49:46.049073 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.049093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.049316 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.049521 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.049522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.049526 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:46.049606 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.049608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.050970 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.050972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.052192 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.052196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.053057 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.053058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.053128 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.053131 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.053182 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.053244 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":45}
I20260619 02:49:46.053355 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.049187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:60175 (local address 127.26.98.1:36779)
0619 02:49:46.049358 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.049360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.049375 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:49:46.049415 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.049416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.049433 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.049513 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:46.049881 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.050775 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.052344 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.052973 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.053057 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.053074 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.053195 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.053281 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:49:46.053452 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.049126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:59157 (local address 127.26.98.3:37185)
0619 02:49:46.049267 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.049271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.049584 (+ 313us) server_negotiation.cc:407] Connection header received
0619 02:49:46.049693 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.049695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.049711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.049828 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:46.050019 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.050859 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.052396 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.053012 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.053065 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.053082 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.053321 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.053373 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 02:49:46.053673 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.049019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37185 (local address 127.26.98.2:59157)
0619 02:49:46.049508 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.049520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.049614 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.049846 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.049847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.049850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:46.049914 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.049916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.051026 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.051027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.052271 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.052275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.053149 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.053150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.053231 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.053237 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.053548 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.053604 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":165,"thread_start_us":246,"threads_started":1}
I20260619 02:49:46.053901 1571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37c4db4a494390a7a9088906a150ce" candidate_uuid: "6ceec07f218e46b0bdad52c30c60430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3777adda73a14d3f8f79616234029d65" is_pre_election: true
I20260619 02:49:46.053911 1844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37c4db4a494390a7a9088906a150ce" candidate_uuid: "6ceec07f218e46b0bdad52c30c60430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62d104c829684c7793489e48278a1018" is_pre_election: true
I20260619 02:49:46.054035 1571 raft_consensus.cc:2454] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ceec07f218e46b0bdad52c30c60430a in term 0.
I20260619 02:49:46.054061 1844 raft_consensus.cc:2454] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ceec07f218e46b0bdad52c30c60430a in term 0.
I20260619 02:49:46.054220 1642 leader_election.cc:302] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3777adda73a14d3f8f79616234029d65, 6ceec07f218e46b0bdad52c30c60430a; no voters:
I20260619 02:49:46.054391 1990 raft_consensus.cc:2781] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:46.054443 1990 raft_consensus.cc:482] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:46.054466 1990 raft_consensus.cc:3037] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.055611 1990 raft_consensus.cc:504] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:46.055804 1990 leader_election.cc:288] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [CANDIDATE]: Term 1 election: Requested vote from peers 62d104c829684c7793489e48278a1018 (127.26.98.3:37185), 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779)
I20260619 02:49:46.056010 1844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37c4db4a494390a7a9088906a150ce" candidate_uuid: "6ceec07f218e46b0bdad52c30c60430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62d104c829684c7793489e48278a1018"
I20260619 02:49:46.056103 1844 raft_consensus.cc:3037] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.056090 1571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37c4db4a494390a7a9088906a150ce" candidate_uuid: "6ceec07f218e46b0bdad52c30c60430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3777adda73a14d3f8f79616234029d65"
I20260619 02:49:46.056202 1571 raft_consensus.cc:3037] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.056936 1844 raft_consensus.cc:2454] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ceec07f218e46b0bdad52c30c60430a in term 1.
I20260619 02:49:46.056993 1571 raft_consensus.cc:2454] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ceec07f218e46b0bdad52c30c60430a in term 1.
I20260619 02:49:46.057174 1643 leader_election.cc:302] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3777adda73a14d3f8f79616234029d65, 62d104c829684c7793489e48278a1018, 6ceec07f218e46b0bdad52c30c60430a; no voters:
I20260619 02:49:46.057294 1990 raft_consensus.cc:2781] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:46.057459 1990 raft_consensus.cc:686] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 1 LEADER]: Becoming Leader. State: Replica: 6ceec07f218e46b0bdad52c30c60430a, State: Running, Role: LEADER
I20260619 02:49:46.057529 1990 consensus_queue.cc:237] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } }
I20260619 02:49:46.057860 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.057743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:54147)
0619 02:49:46.057781 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.057819 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:49:46.058404 1437 catalog_manager.cc:5495] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2). New cstate: current_term: 1 leader_uuid: "6ceec07f218e46b0bdad52c30c60430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ceec07f218e46b0bdad52c30c60430a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:46.074386 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.071702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.26.98.2:40415)
0619 02:49:46.071778 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.071780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.071789 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:46.071848 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.071849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.071878 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.071950 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:46.072111 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.072642 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.073502 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.073710 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.073782 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.073956 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.074099 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.074178 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.074244 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.074279 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:49:46.076229 1571 raft_consensus.cc:1261] T 1e37c4db4a494390a7a9088906a150ce P 3777adda73a14d3f8f79616234029d65 [term 1 FOLLOWER]: Refusing update from remote peer 6ceec07f218e46b0bdad52c30c60430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.076328 1844 raft_consensus.cc:1261] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Refusing update from remote peer 6ceec07f218e46b0bdad52c30c60430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.076733 1990 consensus_queue.cc:1035] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "62d104c829684c7793489e48278a1018" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.077016 1995 consensus_queue.cc:1035] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3777adda73a14d3f8f79616234029d65" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.078176 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.077969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:42381)
0619 02:49:46.078029 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.078074 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:49:46.078725 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.077969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:50301)
0619 02:49:46.078638 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.078684 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635}
I20260619 02:49:46.081403 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.080519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.3:39115)
0619 02:49:46.081303 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.081349 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":761}
I20260619 02:49:46.082033 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.081560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.1:54973)
0619 02:49:46.081951 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.081990 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
I20260619 02:49:46.083164 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.080766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40659 (local address 127.26.98.2:37455)
0619 02:49:46.083083 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.083127 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2292}
I20260619 02:49:46.084261 1437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.084394 1437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.086347 1437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28b079c8bda844f79b2ac1c702946dc3: Sending DeleteTablet for 3 replicas of tablet 1e37c4db4a494390a7a9088906a150ce
I20260619 02:49:46.086802 1688 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e37c4db4a494390a7a9088906a150ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:57076
I20260619 02:49:46.086829 1551 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e37c4db4a494390a7a9088906a150ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:55426
I20260619 02:49:46.086859 1824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e37c4db4a494390a7a9088906a150ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='oryx', principal='oryx/127.26.98.60@KRBTEST.COM'} at 127.0.0.1:37236
I20260619 02:49:46.087455 2008 tablet_replica.cc:330] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: stopping tablet replica
I20260619 02:49:46.087512 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 1398
I20260619 02:49:46.087558 2008 raft_consensus.cc:2229] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:46.087733 2008 raft_consensus.cc:2258] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.088505 2008 ts_tablet_manager.cc:1905] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.090651 2008 ts_tablet_manager.cc:1918] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:46.090729 2008 log.cc:1196] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/1e37c4db4a494390a7a9088906a150ce
I20260619 02:49:46.091001 2008 ts_tablet_manager.cc:1939] T 1e37c4db4a494390a7a9088906a150ce P 6ceec07f218e46b0bdad52c30c60430a: Deleting consensus metadata
I20260619 02:49:46.091414 2010 tablet_replica.cc:330] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: stopping tablet replica
I20260619 02:49:46.091930 2010 raft_consensus.cc:2229] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:46.092058 2010 raft_consensus.cc:2258] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.092170 1423 catalog_manager.cc:4841] TS 6ceec07f218e46b0bdad52c30c60430a (127.26.98.2:40415): tablet 1e37c4db4a494390a7a9088906a150ce (table test-table [id=145b33b0435347349324d453a43605b3]) successfully deleted
I20260619 02:49:46.092674 2010 ts_tablet_manager.cc:1905] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.094995 2010 ts_tablet_manager.cc:1918] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:46.095489 2010 log.cc:1196] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/1e37c4db4a494390a7a9088906a150ce
W20260619 02:49:46.095852 1962 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:46.095428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36779 (local address 127.0.0.1:55438)
0619 02:49:46.095501 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.095514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.095558 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.095780 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36779: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:49:46.095958 2010 ts_tablet_manager.cc:1939] T 1e37c4db4a494390a7a9088906a150ce P 62d104c829684c7793489e48278a1018: Deleting consensus metadata
W20260619 02:49:46.096078 1423 catalog_manager.cc:4584] TS 3777adda73a14d3f8f79616234029d65 (127.26.98.1:36779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e37c4db4a494390a7a9088906a150ce: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36779: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:49:46.096187 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 1625
I20260619 02:49:46.096542 1424 catalog_manager.cc:4841] TS 62d104c829684c7793489e48278a1018 (127.26.98.3:37185): tablet 1e37c4db4a494390a7a9088906a150ce (table test-table [id=145b33b0435347349324d453a43605b3]) successfully deleted
I20260619 02:49:46.102766 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 1761
I20260619 02:49:46.109483 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 1119
I20260619 02:49:46.114650 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 1306
2026-06-19T02:49:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4865 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:46 dist-test-slave-0f9d krb5kdc[2013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:46 dist-test-slave-0f9d krb5kdc[2013](info): set up 2 sockets
Jun 19 02:49:46 dist-test-slave-0f9d krb5kdc[2013](info): commencing operation
krb5kdc: starting...
W20260619 02:49:47.111738 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab.
Entry for principal oryx/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab.
I20260619 02:49:47.251204 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:37581
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab
--principal=oryx/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:37581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:47.346681 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:47.347157 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:47.347210 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:47.348809 2026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:47.348894 2026 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:49:47.348912 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:47.348932 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:47.348946 2026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:47.348961 2026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:47.351063 2026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:37581
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.62.keytab
--principal=oryx/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:47.351420 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:47.351692 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:47.357231 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.357224 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.357224 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.357270 2026 server_base.cc:1034] running on GCE node
I20260619 02:49:47.357733 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:47.358080 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:47.359275 2026 hybrid_clock.cc:648] HybridClock initialized: now 1781837387359254 us; error 39 us; skew 500 ppm
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:47.362494 2026 init.cc:377] Logged in from keytab as oryx/127.26.98.62@KRBTEST.COM (short username oryx)
I20260619 02:49:47.363161 2026 webserver.cc:460] Webserver started at http://127.26.98.62:43675/ using document root <none> and password file <none>
I20260619 02:49:47.363432 2026 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:47.363484 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:47.363641 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:47.364953 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "bae58923d40543d8a1d8138d33252944"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "5a70bb20e44830de5b8ac70b615fa460"
server_key_iv: "91bf21cf43b5ff778e24e23186bd3311"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.365355 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "bae58923d40543d8a1d8138d33252944"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "5a70bb20e44830de5b8ac70b615fa460"
server_key_iv: "91bf21cf43b5ff778e24e23186bd3311"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.367056 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:47.367937 2041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.368153 2026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:47.368242 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "bae58923d40543d8a1d8138d33252944"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "5a70bb20e44830de5b8ac70b615fa460"
server_key_iv: "91bf21cf43b5ff778e24e23186bd3311"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.368326 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:47.381327 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:47.384124 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:47.384328 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:47.389698 2026 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:37581
I20260619 02:49:47.389770 2103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:37581 every 8 connection(s)
I20260619 02:49:47.390130 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:47.390992 2104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:47.393764 2104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Bootstrap starting.
I20260619 02:49:47.394476 2104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:47.394770 2104 log.cc:824] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:47.395620 2104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: No bootstrap required, opened a new log
I20260619 02:49:47.397817 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2026
I20260619 02:49:47.397950 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:47.398197 27016 external_mini_cluster.cc:1428] Setting key 705a910ace621af471a0ed214b758e4a
I20260619 02:49:47.398159 2104 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } }
I20260619 02:49:47.398373 2104 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:47.398424 2104 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae58923d40543d8a1d8138d33252944, State: Initialized, Role: FOLLOWER
I20260619 02:49:47.398598 2104 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } }
I20260619 02:49:47.398692 2104 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:47.398756 2104 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:47.398833 2104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:47.399610 2104 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } }
I20260619 02:49:47.399727 2104 leader_election.cc:302] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae58923d40543d8a1d8138d33252944; no voters:
I20260619 02:49:47.399917 2104 leader_election.cc:288] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:47.399991 2109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:47.400183 2109 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [term 1 LEADER]: Becoming Leader. State: Replica: bae58923d40543d8a1d8138d33252944, State: Running, Role: LEADER
I20260619 02:49:47.400226 2104 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:47.400393 2109 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } }
I20260619 02:49:47.400888 2111 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae58923d40543d8a1d8138d33252944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } } }
I20260619 02:49:47.400923 2110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae58923d40543d8a1d8138d33252944. Latest consensus state: current_term: 1 leader_uuid: "bae58923d40543d8a1d8138d33252944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae58923d40543d8a1d8138d33252944" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 37581 } } }
I20260619 02:49:47.401079 2111 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:47.401100 2110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:47.401299 2118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:47.401923 2118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:47.407714 2118 catalog_manager.cc:1347] Generated new cluster ID: fb5d648983f44a7f97b8d987f046e483
I20260619 02:49:47.407799 2118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:47.411309 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.398795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.26.98.62:37581)
0619 02:49:47.399061 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.399074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.399125 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:49:47.399253 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.399260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.399317 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.399475 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.400315 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.401790 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.402773 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.403409 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.403557 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.406766 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.406775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.406775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.406799 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.409245 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.409785 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.409790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.409790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.409940 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.410133 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.410133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.410134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.410535 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.410739 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.410819 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.411026 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":144,"threads_started":1}
I20260619 02:49:47.424216 2118 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:47.425110 2118 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:47.448680 2118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Generated new TSK 0
I20260619 02:49:47.449028 2118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
I20260619 02:49:47.482313 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:37581
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:47.578759 2130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:47.579095 2130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:47.579140 2130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:47.580677 2130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:47.580760 2130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:47.580796 2130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:47.580822 2130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:47.582867 2130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:47.583305 2130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:47.583588 2130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:47.590487 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.590473 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.590704 2130 server_base.cc:1034] running on GCE node
W20260619 02:49:47.590474 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.591248 2130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:47.591600 2130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:47.592804 2130 hybrid_clock.cc:648] HybridClock initialized: now 1781837387592772 us; error 63 us; skew 500 ppm
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:47.596395 2130 init.cc:377] Logged in from keytab as oryx/127.26.98.1@KRBTEST.COM (short username oryx)
I20260619 02:49:47.597093 2130 webserver.cc:460] Webserver started at http://127.26.98.1:38959/ using document root <none> and password file <none>
I20260619 02:49:47.597368 2130 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:47.597424 2130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:47.597574 2130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:47.599066 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "1438113cbca148818c18b95f53056321"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "7a1030f2de94dfaa9e1e33328a86e52d"
server_key_iv: "29afb6aecd120883bd1efe3e18e360ce"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.599521 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "1438113cbca148818c18b95f53056321"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "7a1030f2de94dfaa9e1e33328a86e52d"
server_key_iv: "29afb6aecd120883bd1efe3e18e360ce"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.601135 2130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:47.602007 2145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.602208 2130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:47.602311 2130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "1438113cbca148818c18b95f53056321"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "7a1030f2de94dfaa9e1e33328a86e52d"
server_key_iv: "29afb6aecd120883bd1efe3e18e360ce"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.602385 2130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:47.619743 2130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:47.622368 2130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:47.622547 2130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:47.623297 2130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:47.623355 2130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.623392 2130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:47.623404 2130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:47.632475 2130 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:32961
I20260619 02:49:47.632665 2260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:32961 every 8 connection(s)
I20260619 02:49:47.633065 2130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:47.635972 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.623808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:48275 (local address 127.26.98.62:37581)
0619 02:49:47.623892 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.623895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.624194 (+ 299us) server_negotiation.cc:407] Connection header received
0619 02:49:47.625084 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.625086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.625103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.625181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.626549 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.627229 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.628466 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.628689 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.628762 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.632296 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.632299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.632299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.632312 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.634393 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.634971 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.634972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.634973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.635043 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.635341 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.635342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.635342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.635471 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.635520 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.635798 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.635888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:49:47.636189 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.623798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.1:48275)
0619 02:49:47.623993 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:47.624056 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:47.624908 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:47.625363 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:47.625369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:47.625974 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:47.626406 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.626417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.627346 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.627348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.628353 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.628356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.628790 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.628791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.628956 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.629667 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:47.629682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:47.632045 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:47.634511 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.634521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.634524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.634842 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.635109 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.635111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.635111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.635280 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.635515 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:47.635519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:47.635678 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 02:49:47.635771 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:49:47.635942 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260619 02:49:47.636778 2261 heartbeater.cc:347] Connected to a master server at 127.26.98.62:37581
I20260619 02:49:47.636898 2261 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:47.637121 2261 heartbeater.cc:511] Master 127.26.98.62:37581 requested a full tablet report, sending...
I20260619 02:49:47.637905 2057 ts_manager.cc:194] Registered new tserver with Master: 1438113cbca148818c18b95f53056321 (127.26.98.1:32961)
I20260619 02:49:47.638672 2057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:48275
I20260619 02:49:47.639842 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2130
I20260619 02:49:47.639955 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:47.640172 27016 external_mini_cluster.cc:1428] Setting key 503a1ad8f4bef580b4341918a0accf07
I20260619 02:49:47.644834 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.638012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.1:44915)
0619 02:49:47.638173 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:47.638185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:47.638265 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:47.638409 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:47.638410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:47.638608 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:47.638687 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.638689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.639593 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.639595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.640217 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.640220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.640626 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.640627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.640682 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.641195 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:47.641200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:47.641550 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:47.643385 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.643387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.643388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.643688 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.643978 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.643978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.643979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.644057 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.644456 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:47.644456 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:47.644517 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:49:47.644590 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:49:47.644736 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260619 02:49:47.644845 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.638090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:44915 (local address 127.26.98.62:37581)
0619 02:49:47.638143 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.638146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.638258 (+ 112us) server_negotiation.cc:407] Connection header received
0619 02:49:47.638327 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.638328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.638339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.638387 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.638763 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.639501 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.640292 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.640547 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.640631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.641643 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.641645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.641645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.641657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.643266 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.643822 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.643823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.643824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.643901 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.644122 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.644123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.644123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.644336 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.644418 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.644623 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.644741 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
I20260619 02:49:47.673969 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:37581
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:47.772022 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:47.772352 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:47.772387 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:47.773897 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:47.773970 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:47.773995 2270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:47.774016 2270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:47.776602 2270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:47.776944 2270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:47.777209 2270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:47.783103 2275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.783103 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.783103 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.783510 2270 server_base.cc:1034] running on GCE node
I20260619 02:49:47.783813 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:47.784192 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:47.785454 2270 hybrid_clock.cc:648] HybridClock initialized: now 1781837387785416 us; error 58 us; skew 500 ppm
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:47.789197 2270 init.cc:377] Logged in from keytab as oryx/127.26.98.2@KRBTEST.COM (short username oryx)
I20260619 02:49:47.789963 2270 webserver.cc:460] Webserver started at http://127.26.98.2:43519/ using document root <none> and password file <none>
I20260619 02:49:47.790259 2270 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:47.790540 2270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:47.790751 2270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:47.792259 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "93b7e90e6f894ac4919d1811ee64b19d"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "0063293b6544b971bbe05c15df27afad"
server_key_iv: "d4230d549c0bcec5d18e0a02c37b3a15"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.792716 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "93b7e90e6f894ac4919d1811ee64b19d"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "0063293b6544b971bbe05c15df27afad"
server_key_iv: "d4230d549c0bcec5d18e0a02c37b3a15"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.794612 2270 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:47.796100 2285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.796490 2270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:47.796664 2270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "93b7e90e6f894ac4919d1811ee64b19d"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "0063293b6544b971bbe05c15df27afad"
server_key_iv: "d4230d549c0bcec5d18e0a02c37b3a15"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.796772 2270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:47.811549 2270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:47.814568 2270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:47.814813 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:47.815519 2270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:47.815575 2270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.815621 2270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:47.815644 2270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.824060 2270 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40591
I20260619 02:49:47.824153 2400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40591 every 8 connection(s)
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:47.824645 2270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:47.828675 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.816144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:51863 (local address 127.26.98.62:37581)
0619 02:49:47.816385 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.816389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.816481 (+ 92us) server_negotiation.cc:407] Connection header received
0619 02:49:47.817124 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.817127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.817147 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.817271 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.818458 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.819346 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.821477 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.821783 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.821889 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.825384 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.825387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.825387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.825397 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.827109 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.827652 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.827654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.827654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.827729 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.827972 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.827972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.827973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.828109 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.828167 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.828504 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.828591 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260619 02:49:47.828924 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.816104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.2:51863)
0619 02:49:47.816308 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:47.816361 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:47.816990 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:47.817376 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:47.817383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:47.817845 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:47.818295 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.818306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.819550 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.819552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.821324 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.821331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.821994 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.821997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.822192 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.823039 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:47.823050 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:47.825237 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:47.827217 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.827228 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.827230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.827541 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.827797 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.827798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.827798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.827917 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.828199 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:47.828207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:47.828372 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 02:49:47.828482 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:49:47.828650 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260619 02:49:47.829442 2401 heartbeater.cc:347] Connected to a master server at 127.26.98.62:37581
I20260619 02:49:47.829540 2401 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:47.829759 2401 heartbeater.cc:511] Master 127.26.98.62:37581 requested a full tablet report, sending...
I20260619 02:49:47.830231 2058 ts_manager.cc:194] Registered new tserver with Master: 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591)
I20260619 02:49:47.830672 2058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:51863
I20260619 02:49:47.831243 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2270
I20260619 02:49:47.831370 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:47.831619 27016 external_mini_cluster.cc:1428] Setting key 2a4903114f6e935b91ca763ff50d8587
I20260619 02:49:47.838404 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.830787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:57065 (local address 127.26.98.62:37581)
0619 02:49:47.830834 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.830837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.830896 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:49:47.830980 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.830980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.830992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.831052 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.831673 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.832419 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.833407 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.833645 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.833719 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.835107 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.835109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.835109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.835119 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.836727 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.837489 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.837491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.837492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.837571 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.837766 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.837767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.837768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.837956 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.838025 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.838184 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.838293 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:47.838452 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.830630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.2:57065)
0619 02:49:47.830811 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:47.830824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:47.830927 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:47.831072 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:47.831073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:47.831329 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:47.831507 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.831511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.832521 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.832523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.833286 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:47.833292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.833756 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.833757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:47.833844 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.834471 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:47.834478 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:47.834994 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:47.836869 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.836871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.836872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.837390 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.837638 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:47.837639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:47.837639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:47.837711 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:47.838026 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:47.838027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:47.838090 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 02:49:47.838153 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:49:47.838342 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
I20260619 02:49:47.862515 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:37581
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:47.961499 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:47.961838 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:47.961872 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:47.963501 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:47.963599 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:47.963629 2410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:47.963649 2410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:47.965899 2410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.98.62:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:47.966253 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:47.966547 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:47.972879 2415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.972919 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.972941 2418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.973383 2410 server_base.cc:1034] running on GCE node
I20260619 02:49:47.973606 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:47.973883 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:47.975032 2410 hybrid_clock.cc:648] HybridClock initialized: now 1781837387974999 us; error 50 us; skew 500 ppm
Jun 19 02:49:47 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:47.978112 2410 init.cc:377] Logged in from keytab as oryx/127.26.98.3@KRBTEST.COM (short username oryx)
I20260619 02:49:47.978856 2410 webserver.cc:460] Webserver started at http://127.26.98.3:39241/ using document root <none> and password file <none>
I20260619 02:49:47.979210 2410 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:47.979290 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:47.979519 2410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:47.980908 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "143656e3fdb743fe9a2b30416491f0c5"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "db3a8dc8f245e9dd94c7ffa40326a95d"
server_key_iv: "375cb52478d98bf35e81fc7402093138"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.981562 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "143656e3fdb743fe9a2b30416491f0c5"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "db3a8dc8f245e9dd94c7ffa40326a95d"
server_key_iv: "375cb52478d98bf35e81fc7402093138"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.983327 2410 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:49:47.984382 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.984643 2410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:47.984748 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "143656e3fdb743fe9a2b30416491f0c5"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-0f9d"
server_key: "db3a8dc8f245e9dd94c7ffa40326a95d"
server_key_iv: "375cb52478d98bf35e81fc7402093138"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.984833 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:48.003494 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:48.006552 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:48.006726 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:48.007536 2410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:48.007598 2410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.007630 2410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:48.007647 2410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:48.015856 2410 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:39465
I20260619 02:49:48.016050 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:39465 every 8 connection(s)
I20260619 02:49:48.016369 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:48.019268 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.008032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:38803 (local address 127.26.98.62:37581)
0619 02:49:48.008112 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.008116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.008334 (+ 218us) server_negotiation.cc:407] Connection header received
0619 02:49:48.009091 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.009092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.009108 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.009159 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.009950 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.010742 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.011979 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.012223 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.012291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.015391 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:48.015393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:48.015394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:48.015405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:48.017228 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.017837 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.017839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.017839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.017917 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.018348 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.018350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.018350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.018528 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:48.018595 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.019040 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.019170 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:48.019520 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.008017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.3:38803)
0619 02:49:48.008189 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.008239 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.008954 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.009241 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.009247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.009605 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:48.009851 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.009857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.010836 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.010837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.011874 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.011878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.012332 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.012333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.012473 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.013196 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:48.013207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:48.015236 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:48.017372 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.017380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.017383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.017685 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.018096 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.018097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.018097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.018232 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.018641 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:48.018647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:48.018830 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.018996 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.019224 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260619 02:49:48.019662 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2410
I20260619 02:49:48.019767 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:48.019989 27016 external_mini_cluster.cc:1428] Setting key f110a7e2d86fc3f7beedd58e290c8377
I20260619 02:49:48.020319 2541 heartbeater.cc:347] Connected to a master server at 127.26.98.62:37581
I20260619 02:49:48.020469 2541 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:48.020771 2541 heartbeater.cc:511] Master 127.26.98.62:37581 requested a full tablet report, sending...
I20260619 02:49:48.021313 2058 ts_manager.cc:194] Registered new tserver with Master: 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:48.021845 2058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:38803
I20260619 02:49:48.028756 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.021673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.26.98.3:37609)
0619 02:49:48.021834 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.021847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.021921 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.022081 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.022082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.022340 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:48.022444 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.022448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.023334 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.023335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.023914 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.023916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.024305 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.024306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.024366 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.025042 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:48.025055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:48.025466 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:48.027302 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.027305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.027305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.027593 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.027920 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.027921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.027922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.028002 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.028372 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:48.028374 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:48.028444 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.028527 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.028648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260619 02:49:48.028808 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.021776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:37609 (local address 127.26.98.62:37581)
0619 02:49:48.021830 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.021832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.021902 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:49:48.022012 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.022013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.022025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.022074 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.022523 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.023187 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.023986 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.024187 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.024267 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.025577 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:48.025579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:48.025580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:48.025588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:48.027138 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.027727 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.027729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.027729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.027809 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.028101 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.028102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.028102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.028257 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:48.028325 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.028599 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.028700 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:48.031915 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:49:48.059705 2108 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:48.052429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.26.98.62:37581)
0619 02:49:48.052584 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.052587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.052597 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:48.052629 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.052630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.052664 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.052732 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.053523 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.054120 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.055033 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.055239 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.055297 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.059591 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46530: BlockingRecv error: recv got EOF from 127.0.0.1:46530 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:48.070004 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.060847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.26.98.62:37581)
0619 02:49:48.060939 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.060942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.061023 (+ 81us) server_negotiation.cc:407] Connection header received
0619 02:49:48.061193 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.061194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.061207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.061262 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.061913 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.062470 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.063403 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.063620 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.063692 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.066643 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:48.066646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:48.066646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:48.066659 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:48.068287 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.069000 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.069002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.069002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.069078 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.069375 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.069377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.069377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.069566 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:48.069668 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.069820 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.069911 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:49:48.072484 2058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:48.074750 2058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:48.084594 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.079252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.26.98.3:39465)
0619 02:49:48.079554 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.079558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.079572 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:48.079632 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.079635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.079654 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.079734 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.080376 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081630 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.083260 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.084060 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.084140 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.084224 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.084356 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.084415 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260619 02:49:48.084789 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.079273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.26.98.2:40591)
0619 02:49:48.079455 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.079461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.079666 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:49:48.079729 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.079731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.079749 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.079834 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.080233 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081181 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.082945 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.083848 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.083969 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.084087 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.084611 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.084677 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260619 02:49:48.084913 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.078930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:39465 (local address 127.0.0.1:35642)
0619 02:49:48.079250 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.079302 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.079415 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.080187 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.080188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.080191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.080289 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.080291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081784 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.081785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.083138 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.083141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.084173 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.084173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.084232 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.084234 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.084294 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.084821 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":536,"spinlock_wait_cycles":1280,"thread_start_us":142,"threads_started":1}
I20260619 02:49:48.084935 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.078930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40591 (local address 127.0.0.1:33186)
0619 02:49:48.079556 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.079566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.079653 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.079946 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.079949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.079952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.080080 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.080083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081295 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.081296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.082784 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.082788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.084016 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.084018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.084169 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.084194 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.084294 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.084399 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":4,"thread_start_us":116,"threads_started":1}
I20260619 02:49:48.085794 2335 tablet_service.cc:1467] Processing CreateTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 (DEFAULT_TABLE table=test-table [id=82e7f0f37753441c9f095cbecf37f214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.085798 2475 tablet_service.cc:1467] Processing CreateTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 (DEFAULT_TABLE table=test-table [id=82e7f0f37753441c9f095cbecf37f214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.086092 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.078869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:32961 (local address 127.0.0.1:39876)
0619 02:49:48.079171 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.079444 (+ 273us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.079496 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.080123 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.080126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.080134 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.080286 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.080290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081784 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.081785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.083160 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.083162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.085801 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.085803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.085882 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.085888 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.085953 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.086006 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":25,"thread_start_us":147,"threads_started":1}
I20260619 02:49:48.086339 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db59c3c7d2d84f7c98a9ac51978fd496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.086338 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db59c3c7d2d84f7c98a9ac51978fd496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.086450 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.079040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.26.98.1:32961)
0619 02:49:48.079842 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.079847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.079856 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:48.079918 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.079920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.079937 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.080083 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.080413 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.081630 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.084284 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.085686 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.085781 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.085874 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.085967 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.086021 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":114,"threads_started":1}
I20260619 02:49:48.087507 2195 tablet_service.cc:1467] Processing CreateTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 (DEFAULT_TABLE table=test-table [id=82e7f0f37753441c9f095cbecf37f214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.088275 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db59c3c7d2d84f7c98a9ac51978fd496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.091293 2569 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Bootstrap starting.
I20260619 02:49:48.092163 2569 tablet_bootstrap.cc:654] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.093034 2569 log.cc:824] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:48.094340 2569 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: No bootstrap required, opened a new log
I20260619 02:49:48.094465 2569 ts_tablet_manager.cc:1397] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:48.094811 2572 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Bootstrap starting.
I20260619 02:49:48.094344 2571 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Bootstrap starting.
I20260619 02:49:48.095867 2572 tablet_bootstrap.cc:654] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.096186 2571 tablet_bootstrap.cc:654] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.096372 2572 log.cc:824] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:48.097116 2569 raft_consensus.cc:348] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.097373 2569 raft_consensus.cc:374] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.097412 2569 raft_consensus.cc:729] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.097626 2569 consensus_queue.cc:260] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.098361 2569 ts_tablet_manager.cc:1428] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 02:49:48.098454 2541 heartbeater.cc:503] Master 127.26.98.62:37581 was elected leader, sending a full tablet report...
I20260619 02:49:48.098946 2571 log.cc:824] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:48.100018 2572 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: No bootstrap required, opened a new log
I20260619 02:49:48.100142 2572 ts_tablet_manager.cc:1397] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:49:48.100155 2571 tablet_bootstrap.cc:492] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: No bootstrap required, opened a new log
I20260619 02:49:48.100222 2571 ts_tablet_manager.cc:1397] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 02:49:48.103672 2571 raft_consensus.cc:348] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.103677 2572 raft_consensus.cc:348] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.103994 2571 raft_consensus.cc:374] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.103994 2572 raft_consensus.cc:374] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.104040 2572 raft_consensus.cc:729] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1438113cbca148818c18b95f53056321, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.104040 2571 raft_consensus.cc:729] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.104215 2571 consensus_queue.cc:260] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.104215 2572 consensus_queue.cc:260] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.104588 2572 ts_tablet_manager.cc:1428] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 02:49:48.104588 2571 ts_tablet_manager.cc:1428] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 02:49:48.104763 2401 heartbeater.cc:503] Master 127.26.98.62:37581 was elected leader, sending a full tablet report...
I20260619 02:49:48.104775 2261 heartbeater.cc:503] Master 127.26.98.62:37581 was elected leader, sending a full tablet report...
W20260619 02:49:48.134086 2262 tablet.cc:2369] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:48.168114 2574 raft_consensus.cc:482] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:48.168313 2574 raft_consensus.cc:504] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.168805 2574 leader_election.cc:288] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 1438113cbca148818c18b95f53056321 (127.26.98.1:32961)
I20260619 02:49:48.172290 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.169120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:45477 (local address 127.26.98.1:32961)
0619 02:49:48.169214 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.169216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.169232 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:49:48.169271 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.169272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.169286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.169345 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.169719 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.170275 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.171469 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.171912 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.171985 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.172000 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.172150 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.172204 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:49:48.172291 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.169060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:32961 (local address 127.26.98.3:45477)
0619 02:49:48.169138 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.169151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.169207 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.169355 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.169356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.169359 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.169481 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.169485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.170401 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.170402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.171357 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.171361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.171992 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.171993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.172076 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.172080 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.172132 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.172187 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:49:48.172783 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db59c3c7d2d84f7c98a9ac51978fd496" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321" is_pre_election: true
I20260619 02:49:48.173027 2215 raft_consensus.cc:2454] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 0.
I20260619 02:49:48.173348 2428 leader_election.cc:302] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:48.173559 2574 raft_consensus.cc:2781] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:48.173645 2574 raft_consensus.cc:482] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:48.173684 2574 raft_consensus.cc:3037] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.174120 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.169322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:44841 (local address 127.26.98.2:40591)
0619 02:49:48.169515 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.169518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.169545 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:49:48.169772 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.169774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.169786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.169829 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.169984 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.171488 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.173157 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.173724 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.173807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.173821 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.174013 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.174059 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:48.174142 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.169238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40591 (local address 127.26.98.3:44841)
0619 02:49:48.169492 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.169503 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.169682 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.169841 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.169842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.169844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.169923 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.169925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.171740 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.171741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.173025 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.173029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.173828 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.173829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.173903 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.173908 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.173973 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.174025 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":117,"threads_started":1}
I20260619 02:49:48.174670 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db59c3c7d2d84f7c98a9ac51978fd496" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d" is_pre_election: true
I20260619 02:49:48.174918 2355 raft_consensus.cc:2454] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 0.
I20260619 02:49:48.175287 2574 raft_consensus.cc:504] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.175448 2574 leader_election.cc:288] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 1438113cbca148818c18b95f53056321 (127.26.98.1:32961)
I20260619 02:49:48.175626 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db59c3c7d2d84f7c98a9ac51978fd496" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d"
I20260619 02:49:48.175719 2355 raft_consensus.cc:3037] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.175736 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db59c3c7d2d84f7c98a9ac51978fd496" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321"
I20260619 02:49:48.175840 2215 raft_consensus.cc:3037] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.176847 2215 raft_consensus.cc:2454] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 1.
I20260619 02:49:48.176854 2355 raft_consensus.cc:2454] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 1.
I20260619 02:49:48.177048 2428 leader_election.cc:302] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:48.177191 2574 raft_consensus.cc:2781] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:48.177551 2574 raft_consensus.cc:686] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Running, Role: LEADER
I20260619 02:49:48.177718 2574 consensus_queue.cc:237] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } }
I20260619 02:49:48.179948 2058 catalog_manager.cc:5495] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "143656e3fdb743fe9a2b30416491f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:48.206904 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.203426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.26.98.3:39465)
0619 02:49:48.203484 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.203486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.203499 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:49:48.203557 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.203559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.203591 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.203706 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.204450 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.205019 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.205924 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.206191 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.206259 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.206467 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.206616 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.206715 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.206736 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.206765 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:48.209350 2215 raft_consensus.cc:1261] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Refusing update from remote peer 143656e3fdb743fe9a2b30416491f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.209350 2355 raft_consensus.cc:1261] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Refusing update from remote peer 143656e3fdb743fe9a2b30416491f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.209781 2574 consensus_queue.cc:1035] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.209960 2579 consensus_queue.cc:1035] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.213408 2583 mvcc.cc:205] Tried to move back new op lower bound from 7298405942101864448 to 7298405941976883200. Current Snapshot: MvccSnapshot[applied={T|T < 7298405942101864448}]
I20260619 02:49:48.213860 2584 mvcc.cc:205] Tried to move back new op lower bound from 7298405942101864448 to 7298405941976883200. Current Snapshot: MvccSnapshot[applied={T|T < 7298405942101864448}]
I20260619 02:49:48.216760 2057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.216920 2057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.218603 2057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Sending DeleteTablet for 3 replicas of tablet db59c3c7d2d84f7c98a9ac51978fd496
I20260619 02:49:48.219214 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:35642
I20260619 02:49:48.219211 2195 tablet_service.cc:1514] Processing DeleteTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:39876
I20260619 02:49:48.219511 2335 tablet_service.cc:1514] Processing DeleteTablet for tablet db59c3c7d2d84f7c98a9ac51978fd496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260619 02:49:48.219594 2591 tablet_replica.cc:330] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: stopping tablet replica
I20260619 02:49:48.219602 2057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:49:48.219736 2591 raft_consensus.cc:2229] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 02:49:48.219801 2057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:48.219885 2591 raft_consensus.cc:2258] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.220513 2591 ts_tablet_manager.cc:1905] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.221692 2195 tablet_service.cc:1467] Processing CreateTablet for tablet 51e4204a9730434c82046b700ebe80a9 (DEFAULT_TABLE table=test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.221712 2335 tablet_service.cc:1467] Processing CreateTablet for tablet 51e4204a9730434c82046b700ebe80a9 (DEFAULT_TABLE table=test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.221841 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e4204a9730434c82046b700ebe80a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.221841 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e4204a9730434c82046b700ebe80a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.223088 2591 ts_tablet_manager.cc:1918] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.223181 2591 log.cc:1196] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/db59c3c7d2d84f7c98a9ac51978fd496
I20260619 02:49:48.223483 2591 ts_tablet_manager.cc:1939] T db59c3c7d2d84f7c98a9ac51978fd496 P 1438113cbca148818c18b95f53056321: Deleting consensus metadata
I20260619 02:49:48.224499 2571 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Bootstrap starting.
I20260619 02:49:48.224974 2571 tablet_bootstrap.cc:654] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.225378 2572 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Bootstrap starting.
I20260619 02:49:48.225777 2572 tablet_bootstrap.cc:654] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.226003 2571 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: No bootstrap required, opened a new log
I20260619 02:49:48.226066 2571 ts_tablet_manager.cc:1397] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:48.226265 2571 raft_consensus.cc:348] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.226378 2571 raft_consensus.cc:374] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.226398 2571 raft_consensus.cc:729] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.226444 2571 consensus_queue.cc:260] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.226434 2044 catalog_manager.cc:4841] TS 1438113cbca148818c18b95f53056321 (127.26.98.1:32961): tablet db59c3c7d2d84f7c98a9ac51978fd496 (table test-table [id=82e7f0f37753441c9f095cbecf37f214]) successfully deleted
I20260619 02:49:48.226645 2571 ts_tablet_manager.cc:1428] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.227104 2475 tablet_service.cc:1467] Processing CreateTablet for tablet 51e4204a9730434c82046b700ebe80a9 (DEFAULT_TABLE table=test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.227295 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e4204a9730434c82046b700ebe80a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.227491 2593 tablet_replica.cc:330] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: stopping tablet replica
I20260619 02:49:48.227646 2593 raft_consensus.cc:2229] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.227799 2593 raft_consensus.cc:2258] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.228513 2572 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: No bootstrap required, opened a new log
I20260619 02:49:48.228588 2572 ts_tablet_manager.cc:1397] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:48.229036 2569 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Bootstrap starting.
I20260619 02:49:48.229049 2572 raft_consensus.cc:348] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.229154 2572 raft_consensus.cc:374] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.229173 2572 raft_consensus.cc:729] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1438113cbca148818c18b95f53056321, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.229254 2572 consensus_queue.cc:260] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.229383 2572 ts_tablet_manager.cc:1428] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.229519 2592 tablet_replica.cc:330] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: stopping tablet replica
I20260619 02:49:48.229600 2569 tablet_bootstrap.cc:654] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.229673 2592 raft_consensus.cc:2229] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:48.229820 2592 raft_consensus.cc:2258] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.230370 2592 ts_tablet_manager.cc:1905] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.230374 2593 ts_tablet_manager.cc:1905] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.230834 2569 tablet_bootstrap.cc:492] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: No bootstrap required, opened a new log
I20260619 02:49:48.230927 2569 ts_tablet_manager.cc:1397] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:49:48.231103 2569 raft_consensus.cc:348] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.231180 2569 raft_consensus.cc:374] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.231199 2569 raft_consensus.cc:729] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.231283 2569 consensus_queue.cc:260] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.231473 2569 ts_tablet_manager.cc:1428] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.232551 2592 ts_tablet_manager.cc:1918] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.232584 2593 ts_tablet_manager.cc:1918] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.232641 2592 log.cc:1196] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/db59c3c7d2d84f7c98a9ac51978fd496
I20260619 02:49:48.232669 2593 log.cc:1196] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/db59c3c7d2d84f7c98a9ac51978fd496
I20260619 02:49:48.232980 2592 ts_tablet_manager.cc:1939] T db59c3c7d2d84f7c98a9ac51978fd496 P 143656e3fdb743fe9a2b30416491f0c5: Deleting consensus metadata
I20260619 02:49:48.232978 2593 ts_tablet_manager.cc:1939] T db59c3c7d2d84f7c98a9ac51978fd496 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting consensus metadata
I20260619 02:49:48.233538 2045 catalog_manager.cc:4841] TS 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465): tablet db59c3c7d2d84f7c98a9ac51978fd496 (table test-table [id=82e7f0f37753441c9f095cbecf37f214]) successfully deleted
I20260619 02:49:48.233583 2042 catalog_manager.cc:4841] TS 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591): tablet db59c3c7d2d84f7c98a9ac51978fd496 (table test-table [id=82e7f0f37753441c9f095cbecf37f214]) successfully deleted
W20260619 02:49:48.267241 2542 tablet.cc:2369] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:48.325539 2402 tablet.cc:2369] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:48.348798 2576 raft_consensus.cc:482] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:48.348937 2576 raft_consensus.cc:504] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.349344 2576 leader_election.cc:288] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:48.353433 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.349840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:39063 (local address 127.26.98.3:39465)
0619 02:49:48.349897 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.349900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.349942 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:49:48.350061 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.350062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.350080 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.350110 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.350325 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.351180 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.352446 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.352975 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.353065 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.353081 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.353288 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.353333 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:48.353461 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.349580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:39465 (local address 127.26.98.1:39063)
0619 02:49:48.349865 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.349878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.349968 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.350187 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.350188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.350190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.350260 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.350262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.351309 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.351310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.352310 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.352314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.353101 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.353102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.353194 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.353200 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.353265 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.353322 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":28,"thread_start_us":79,"threads_started":1}
I20260619 02:49:48.353796 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.349777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:56017 (local address 127.26.98.2:40591)
0619 02:49:48.349833 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.349836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.350006 (+ 170us) server_negotiation.cc:407] Connection header received
0619 02:49:48.350061 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.350062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.350076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.350134 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.350357 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.351180 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.352785 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.353308 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.353382 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.353394 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.353645 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.353703 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:49:48.354084 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.349635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40591 (local address 127.26.98.1:56017)
0619 02:49:48.349920 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.349930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.350009 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.350146 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.350147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.350149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.350263 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.350265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.351282 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.351284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.352651 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.352655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.353441 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.353442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.353542 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.353546 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.353910 (+ 364us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.353971 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":49,"thread_start_us":104,"threads_started":1}
I20260619 02:49:48.354207 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e4204a9730434c82046b700ebe80a9" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5" is_pre_election: true
I20260619 02:49:48.354379 2495 raft_consensus.cc:2454] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 0.
I20260619 02:49:48.354395 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e4204a9730434c82046b700ebe80a9" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d" is_pre_election: true
I20260619 02:49:48.354560 2355 raft_consensus.cc:2454] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 0.
I20260619 02:49:48.354764 2149 leader_election.cc:302] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:48.354996 2576 raft_consensus.cc:2781] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:48.355041 2576 raft_consensus.cc:482] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:48.355055 2576 raft_consensus.cc:3037] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.355855 2576 raft_consensus.cc:504] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.356003 2576 leader_election.cc:288] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 election: Requested vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:48.356220 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e4204a9730434c82046b700ebe80a9" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d"
I20260619 02:49:48.356281 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e4204a9730434c82046b700ebe80a9" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5"
I20260619 02:49:48.356324 2355 raft_consensus.cc:3037] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.356362 2495 raft_consensus.cc:3037] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.357101 2495 raft_consensus.cc:2454] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 1.
I20260619 02:49:48.357317 2149 leader_election.cc:302] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:48.357411 2576 raft_consensus.cc:2781] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:48.357409 2355 raft_consensus.cc:2454] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 1.
I20260619 02:49:48.357609 2576 raft_consensus.cc:686] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 1 LEADER]: Becoming Leader. State: Replica: 1438113cbca148818c18b95f53056321, State: Running, Role: LEADER
I20260619 02:49:48.357695 2576 consensus_queue.cc:237] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.358853 2057 catalog_manager.cc:5495] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1438113cbca148818c18b95f53056321 (127.26.98.1). New cstate: current_term: 1 leader_uuid: "1438113cbca148818c18b95f53056321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.62@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:48.419297 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.412015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.26.98.62:37581)
0619 02:49:48.412089 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.412091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.412268 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:49:48.412338 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.412339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.412351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.412404 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.412963 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.413486 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.414179 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.414369 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.414426 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.416682 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:48.416684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:48.416684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:48.416692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:48.418063 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.418475 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.418476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.418477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.418532 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.418690 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.418690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.418691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.418815 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:48.418874 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.419063 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.419160 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:48.419378 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.411982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:37581 (local address 127.0.0.1:46546)
0619 02:49:48.412184 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.412194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.412279 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.412415 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.412416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.412815 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:48.412875 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.412878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.413565 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.413566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.414104 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.414105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.414431 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.414432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.414477 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.415086 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:48.415091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:48.416580 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:48.418156 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.418167 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.418170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.418398 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.418586 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:48.418587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:48.418587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:48.418641 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:48.418872 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:48.418924 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:48.418977 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.419033 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.419229 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":75,"thread_start_us":130,"threads_started":1}
I20260619 02:49:48.421974 2057 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:46546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:49:48.422245 2057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:48.423854 2195 tablet_service.cc:1467] Processing CreateTablet for tablet 03b879f60a934b1e9362be7f1a149835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2177bd397b614b0ba682a962e1457448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:48.423951 2475 tablet_service.cc:1467] Processing CreateTablet for tablet 03b879f60a934b1e9362be7f1a149835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2177bd397b614b0ba682a962e1457448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:48.423934 2335 tablet_service.cc:1467] Processing CreateTablet for tablet 03b879f60a934b1e9362be7f1a149835 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2177bd397b614b0ba682a962e1457448]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:48.424068 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b879f60a934b1e9362be7f1a149835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.424088 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b879f60a934b1e9362be7f1a149835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.424108 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b879f60a934b1e9362be7f1a149835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.425457 2571 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: Bootstrap starting.
I20260619 02:49:48.425664 2569 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: Bootstrap starting.
I20260619 02:49:48.425958 2571 tablet_bootstrap.cc:654] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.425993 2569 tablet_bootstrap.cc:654] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.426302 2572 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: Bootstrap starting.
I20260619 02:49:48.426637 2569 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: No bootstrap required, opened a new log
I20260619 02:49:48.426689 2572 tablet_bootstrap.cc:654] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.426680 2569 ts_tablet_manager.cc:1397] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.426828 2569 raft_consensus.cc:348] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.426945 2569 raft_consensus.cc:374] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.426966 2569 raft_consensus.cc:729] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.427021 2569 consensus_queue.cc:260] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.427126 2569 ts_tablet_manager.cc:1428] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:48.427145 2574 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.427230 2574 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.427359 2571 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: No bootstrap required, opened a new log
I20260619 02:49:48.427408 2571 ts_tablet_manager.cc:1397] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:48.427564 2571 raft_consensus.cc:348] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.427618 2571 raft_consensus.cc:374] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.427631 2571 raft_consensus.cc:729] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.427677 2571 consensus_queue.cc:260] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.427788 2571 ts_tablet_manager.cc:1428] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.427851 2577 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.427943 2577 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.428213 2572 tablet_bootstrap.cc:492] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: No bootstrap required, opened a new log
I20260619 02:49:48.428269 2572 ts_tablet_manager.cc:1397] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:48.428417 2572 raft_consensus.cc:348] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.428474 2572 raft_consensus.cc:374] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.428491 2572 raft_consensus.cc:729] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1438113cbca148818c18b95f53056321, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.428532 2572 consensus_queue.cc:260] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.428624 2572 ts_tablet_manager.cc:1428] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.428695 2599 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.428817 2599 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.484094 2577 raft_consensus.cc:482] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:48.484259 2577 raft_consensus.cc:504] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.484730 2577 leader_election.cc:288] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1438113cbca148818c18b95f53056321 (127.26.98.1:32961), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:48.488010 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.484974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:34351 (local address 127.26.98.3:39465)
0619 02:49:48.485029 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.485031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.485139 (+ 108us) server_negotiation.cc:407] Connection header received
0619 02:49:48.485189 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.485190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.485206 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.485264 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.485421 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.485956 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487190 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.487623 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.487706 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.487874 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.487921 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:48.488032 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.485003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:52055 (local address 127.26.98.1:32961)
0619 02:49:48.485060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.485063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.485116 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:49:48.485187 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.485188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.485198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.485251 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:48.485435 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.485944 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487191 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.487621 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487692 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.487706 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.487875 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.487952 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:48.488065 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.484907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:32961 (local address 127.26.98.2:52055)
0619 02:49:48.485048 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.485060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.485120 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.485286 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.485287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.485290 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.485370 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.485371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.486040 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.486042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.487097 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.487099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.487705 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.487806 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.487812 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.487864 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.487914 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":64,"thread_start_us":73,"threads_started":1}
I20260619 02:49:48.488073 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.484906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:39465 (local address 127.26.98.2:34351)
0619 02:49:48.485056 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.485066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.485134 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.485273 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.485274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.485276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:48.485357 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.485359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.486041 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.486042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.487085 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.487088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.487706 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.487708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.487783 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.487787 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.487843 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.487903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":52,"thread_start_us":77,"threads_started":1}
I20260619 02:49:48.488663 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b879f60a934b1e9362be7f1a149835" candidate_uuid: "93b7e90e6f894ac4919d1811ee64b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321" is_pre_election: true
I20260619 02:49:48.488649 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b879f60a934b1e9362be7f1a149835" candidate_uuid: "93b7e90e6f894ac4919d1811ee64b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5" is_pre_election: true
I20260619 02:49:48.488790 2215 raft_consensus.cc:2454] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b7e90e6f894ac4919d1811ee64b19d in term 0.
I20260619 02:49:48.488791 2495 raft_consensus.cc:2454] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b7e90e6f894ac4919d1811ee64b19d in term 0.
I20260619 02:49:48.489070 2289 leader_election.cc:302] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 93b7e90e6f894ac4919d1811ee64b19d; no voters:
I20260619 02:49:48.489464 2577 raft_consensus.cc:2781] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:48.489532 2577 raft_consensus.cc:482] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:48.489555 2577 raft_consensus.cc:3037] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.490721 2577 raft_consensus.cc:504] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.490994 2577 leader_election.cc:288] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [CANDIDATE]: Term 1 election: Requested vote from peers 1438113cbca148818c18b95f53056321 (127.26.98.1:32961), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:48.491329 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b879f60a934b1e9362be7f1a149835" candidate_uuid: "93b7e90e6f894ac4919d1811ee64b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5"
I20260619 02:49:48.491392 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b879f60a934b1e9362be7f1a149835" candidate_uuid: "93b7e90e6f894ac4919d1811ee64b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321"
I20260619 02:49:48.491500 2215 raft_consensus.cc:3037] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.491501 2495 raft_consensus.cc:3037] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.492492 2215 raft_consensus.cc:2454] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b7e90e6f894ac4919d1811ee64b19d in term 1.
I20260619 02:49:48.492492 2495 raft_consensus.cc:2454] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b7e90e6f894ac4919d1811ee64b19d in term 1.
I20260619 02:49:48.492780 2289 leader_election.cc:302] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 93b7e90e6f894ac4919d1811ee64b19d; no voters:
I20260619 02:49:48.492893 2577 raft_consensus.cc:2781] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:48.493064 2577 raft_consensus.cc:686] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 LEADER]: Becoming Leader. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Running, Role: LEADER
I20260619 02:49:48.493148 2577 consensus_queue.cc:237] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.493628 2612 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: TxnStatusTablet state changed. Reason: New leader 93b7e90e6f894ac4919d1811ee64b19d. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.493746 2612 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:48.493942 2614 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:49:48.494072 2614 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:49:48.494171 2058 catalog_manager.cc:5495] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d reported cstate change: term changed from 0 to 1, leader changed from <none> to 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2). New cstate: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:48.507278 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.503657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40591 (local address 127.0.0.1:33192)
0619 02:49:48.503781 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.503798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.503874 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.504095 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.504096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.504098 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:48.504168 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.504172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.505170 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.505172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.505929 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.505934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.506501 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.506502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.506579 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.506607 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.507045 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.507055 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.507105 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.507196 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:49:48.507246 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.503768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.26.98.2:40591)
0619 02:49:48.503874 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.503877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.503892 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:49:48.503969 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.503970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.503997 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.504086 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.504247 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.505008 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.506055 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.506395 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.506496 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.506781 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.506973 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.507065 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.507111 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.507137 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:49:48.508859 2495 raft_consensus.cc:1261] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Refusing update from remote peer 93b7e90e6f894ac4919d1811ee64b19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.508842 2215 raft_consensus.cc:1261] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Refusing update from remote peer 93b7e90e6f894ac4919d1811ee64b19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.509222 2577 consensus_queue.cc:1035] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.509447 2612 consensus_queue.cc:1035] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.510176 2599 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: TxnStatusTablet state changed. Reason: New leader 93b7e90e6f894ac4919d1811ee64b19d. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.510298 2599 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.510393 2574 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: TxnStatusTablet state changed. Reason: New leader 93b7e90e6f894ac4919d1811ee64b19d. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.510497 2574 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.510936 2612 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.511040 2612 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:48.511606 2599 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.511709 2599 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 1438113cbca148818c18b95f53056321: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.511890 2612 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.511972 2612 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 93b7e90e6f894ac4919d1811ee64b19d: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:48.512310 2582 mvcc.cc:205] Tried to move back new op lower bound from 7298405943328956416 to 7298405943268458496. Current Snapshot: MvccSnapshot[applied={T|T < 7298405943328956416}]
I20260619 02:49:48.512488 2574 tablet_replica.cc:439] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93b7e90e6f894ac4919d1811ee64b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } } }
I20260619 02:49:48.512568 2574 tablet_replica.cc:442] T 03b879f60a934b1e9362be7f1a149835 P 143656e3fdb743fe9a2b30416491f0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:48.516160 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.513111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.26.98.1:32961)
0619 02:49:48.513158 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.513160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.513168 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:48.513211 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.513212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.513235 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.513308 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.513457 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.513971 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.515186 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.515412 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.515480 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.515774 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.515912 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.515981 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.516057 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.516081 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:49:48.523283 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.519960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:47965 (local address 127.26.98.2:40591)
0619 02:49:48.520018 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.520020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.520127 (+ 107us) server_negotiation.cc:407] Connection header received
0619 02:49:48.520184 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.520185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.520197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.520263 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.520524 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.521096 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.522394 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.522622 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.522695 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.522970 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.523010 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.523060 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.523131 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.523193 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:48.523329 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.519858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40591 (local address 127.26.98.1:47965)
0619 02:49:48.520055 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.520066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.520128 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.520326 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.520327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.520330 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:48.520436 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.520439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.521204 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.521205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.522258 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.522263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.522710 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.522710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.522789 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.522822 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.523076 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.523078 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.523125 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.523191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260619 02:49:48.526477 2495 raft_consensus.cc:1261] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Refusing update from remote peer 1438113cbca148818c18b95f53056321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.526517 2355 raft_consensus.cc:1261] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Refusing update from remote peer 1438113cbca148818c18b95f53056321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.526821 2576 consensus_queue.cc:1035] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.527048 2599 consensus_queue.cc:1035] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.539920 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.536218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:32961 (local address 127.26.98.2:58905)
0619 02:49:48.536418 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.536428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:48.536496 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:48.536663 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:48.536664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:48.536666 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:48.536765 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.536767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.537926 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.537929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.538980 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:48.538985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.539399 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.539400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:48.539466 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.539499 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.539706 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.539709 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:48.539761 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:49:48.539817 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
I20260619 02:49:48.539906 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.536279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:58905 (local address 127.26.98.1:32961)
0619 02:49:48.536327 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.536330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.536488 (+ 158us) server_negotiation.cc:407] Connection header received
0619 02:49:48.536590 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.536591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.536604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.536656 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.536860 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.537729 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.539078 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.539303 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.539365 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.539596 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.539654 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.539706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.539766 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.539796 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:48.551986 2058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.552187 2058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.553153 2058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Sending DeleteTablet for 3 replicas of tablet 51e4204a9730434c82046b700ebe80a9
I20260619 02:49:48.553568 2195 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e4204a9730434c82046b700ebe80a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:39876
I20260619 02:49:48.553691 2591 tablet_replica.cc:330] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: stopping tablet replica
I20260619 02:49:48.553654 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e4204a9730434c82046b700ebe80a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:35642
I20260619 02:49:48.553761 2591 raft_consensus.cc:2229] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:48.553789 2592 tablet_replica.cc:330] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: stopping tablet replica
I20260619 02:49:48.553855 2592 raft_consensus.cc:2229] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.553879 2591 raft_consensus.cc:2258] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.553897 2592 raft_consensus.cc:2258] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.554502 2591 ts_tablet_manager.cc:1905] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.556835 2591 ts_tablet_manager.cc:1918] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:48.556931 2591 log.cc:1196] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/51e4204a9730434c82046b700ebe80a9
I20260619 02:49:48.557173 2591 ts_tablet_manager.cc:1939] T 51e4204a9730434c82046b700ebe80a9 P 1438113cbca148818c18b95f53056321: Deleting consensus metadata
I20260619 02:49:48.557761 2335 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e4204a9730434c82046b700ebe80a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260619 02:49:48.557830 2044 catalog_manager.cc:4841] TS 1438113cbca148818c18b95f53056321 (127.26.98.1:32961): tablet 51e4204a9730434c82046b700ebe80a9 (table test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]) successfully deleted
I20260619 02:49:48.557916 2593 tablet_replica.cc:330] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: stopping tablet replica
I20260619 02:49:48.557976 2593 raft_consensus.cc:2229] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.558028 2593 raft_consensus.cc:2258] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.558530 2593 ts_tablet_manager.cc:1905] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.560534 2593 ts_tablet_manager.cc:1918] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:48.560600 2593 log.cc:1196] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/51e4204a9730434c82046b700ebe80a9
I20260619 02:49:48.560786 2593 ts_tablet_manager.cc:1939] T 51e4204a9730434c82046b700ebe80a9 P 93b7e90e6f894ac4919d1811ee64b19d: Deleting consensus metadata
I20260619 02:49:48.561233 2042 catalog_manager.cc:4841] TS 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591): tablet 51e4204a9730434c82046b700ebe80a9 (table test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]) successfully deleted
I20260619 02:49:48.567282 2592 ts_tablet_manager.cc:1905] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.569185 2592 ts_tablet_manager.cc:1918] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:48.569255 2592 log.cc:1196] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/51e4204a9730434c82046b700ebe80a9
I20260619 02:49:48.569465 2592 ts_tablet_manager.cc:1939] T 51e4204a9730434c82046b700ebe80a9 P 143656e3fdb743fe9a2b30416491f0c5: Deleting consensus metadata
I20260619 02:49:48.570132 2045 catalog_manager.cc:4841] TS 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465): tablet 51e4204a9730434c82046b700ebe80a9 (table test-table [id=50ee049a4ab44e35b3a5f0ea6d996c27]) successfully deleted
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:49:48.589937 2108 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:48.581556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.26.98.62:37581)
0619 02:49:48.581623 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.581627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.581859 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:49:48.581949 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.581950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.581965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.582021 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.582847 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.583681 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.585063 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.585407 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.585503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.589843 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46556: BlockingRecv error: recv got EOF from 127.0.0.1:46556 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 02:49:48 dist-test-slave-0f9d krb5kdc[2013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM
I20260619 02:49:48.605330 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.591556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.26.98.62:37581)
0619 02:49:48.591633 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.591635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.595187 (+ 3552us) server_negotiation.cc:407] Connection header received
0619 02:49:48.595257 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.595259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.595281 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.595440 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:48.596274 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.597139 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.598323 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.598683 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.598761 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.602172 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:48.602174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:48.602174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:48.602185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:48.604003 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.604451 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.604452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.604453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.604522 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:48.604738 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:48.604739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:48.604739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:48.604905 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:48.605019 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.605130 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.605222 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:49:48.607785 2058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:48.608031 2058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:48.609645 2195 tablet_service.cc:1467] Processing CreateTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f (DEFAULT_TABLE table=test-table [id=398f5b77348345f28449cf77e8eb6ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.609843 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3dec30dba5d4fb39b4c3895acb12d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.611181 2475 tablet_service.cc:1467] Processing CreateTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f (DEFAULT_TABLE table=test-table [id=398f5b77348345f28449cf77e8eb6ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.611333 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3dec30dba5d4fb39b4c3895acb12d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.611712 2572 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Bootstrap starting.
I20260619 02:49:48.612231 2572 tablet_bootstrap.cc:654] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.612828 2569 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Bootstrap starting.
I20260619 02:49:48.613080 2572 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: No bootstrap required, opened a new log
I20260619 02:49:48.613130 2572 ts_tablet_manager.cc:1397] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.613229 2569 tablet_bootstrap.cc:654] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.613292 2572 raft_consensus.cc:348] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.613374 2572 raft_consensus.cc:374] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.613394 2572 raft_consensus.cc:729] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1438113cbca148818c18b95f53056321, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.613441 2572 consensus_queue.cc:260] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.613545 2572 ts_tablet_manager.cc:1428] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.613755 2335 tablet_service.cc:1467] Processing CreateTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f (DEFAULT_TABLE table=test-table [id=398f5b77348345f28449cf77e8eb6ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.613900 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3dec30dba5d4fb39b4c3895acb12d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.615419 2571 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Bootstrap starting.
I20260619 02:49:48.615895 2571 tablet_bootstrap.cc:654] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.616503 2569 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: No bootstrap required, opened a new log
I20260619 02:49:48.616578 2569 ts_tablet_manager.cc:1397] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 02:49:48.616743 2569 raft_consensus.cc:348] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.616837 2569 raft_consensus.cc:374] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.616858 2569 raft_consensus.cc:729] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.616911 2569 consensus_queue.cc:260] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.617012 2569 ts_tablet_manager.cc:1428] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:48.618211 2571 tablet_bootstrap.cc:492] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: No bootstrap required, opened a new log
I20260619 02:49:48.618283 2571 ts_tablet_manager.cc:1397] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:48.618462 2571 raft_consensus.cc:348] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.618580 2571 raft_consensus.cc:374] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.618600 2571 raft_consensus.cc:729] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.618649 2571 consensus_queue.cc:260] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.618742 2571 ts_tablet_manager.cc:1428] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:48.826023 2574 raft_consensus.cc:482] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:48.826176 2574 raft_consensus.cc:504] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.826382 2574 leader_election.cc:288] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1438113cbca148818c18b95f53056321 (127.26.98.1:32961), 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591)
I20260619 02:49:48.826722 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3dec30dba5d4fb39b4c3895acb12d7f" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321" is_pre_election: true
I20260619 02:49:48.826714 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3dec30dba5d4fb39b4c3895acb12d7f" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d" is_pre_election: true
I20260619 02:49:48.826838 2215 raft_consensus.cc:2454] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 0.
I20260619 02:49:48.826836 2355 raft_consensus.cc:2454] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 0.
I20260619 02:49:48.827050 2428 leader_election.cc:302] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321, 93b7e90e6f894ac4919d1811ee64b19d; no voters:
I20260619 02:49:48.827190 2574 raft_consensus.cc:2781] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:48.827244 2574 raft_consensus.cc:482] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:48.827263 2574 raft_consensus.cc:3037] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.828315 2574 raft_consensus.cc:504] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.828471 2574 leader_election.cc:288] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 1438113cbca148818c18b95f53056321 (127.26.98.1:32961), 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591)
I20260619 02:49:48.828720 2215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3dec30dba5d4fb39b4c3895acb12d7f" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1438113cbca148818c18b95f53056321"
I20260619 02:49:48.828717 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3dec30dba5d4fb39b4c3895acb12d7f" candidate_uuid: "143656e3fdb743fe9a2b30416491f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d"
I20260619 02:49:48.828842 2355 raft_consensus.cc:3037] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.828842 2215 raft_consensus.cc:3037] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:48.829792 2355 raft_consensus.cc:2454] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 1.
I20260619 02:49:48.829795 2215 raft_consensus.cc:2454] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 143656e3fdb743fe9a2b30416491f0c5 in term 1.
I20260619 02:49:48.829990 2428 leader_election.cc:302] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321, 93b7e90e6f894ac4919d1811ee64b19d; no voters:
I20260619 02:49:48.830116 2574 raft_consensus.cc:2781] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:48.830276 2574 raft_consensus.cc:686] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Running, Role: LEADER
I20260619 02:49:48.830343 2574 consensus_queue.cc:237] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.831219 2058 catalog_manager.cc:5495] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "143656e3fdb743fe9a2b30416491f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:48.853829 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.851271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.26.98.3:39465)
0619 02:49:48.851367 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:48.851370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:48.851382 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:48.851416 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:48.851418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:48.851430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:48.851491 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:48.851651 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.852214 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.853082 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:48.853286 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:48.853342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:48.853523 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:48.853566 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:48.853622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:48.853717 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:49:48.853741 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:49:48.854859 2215 raft_consensus.cc:1261] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Refusing update from remote peer 143656e3fdb743fe9a2b30416491f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.854851 2355 raft_consensus.cc:1261] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Refusing update from remote peer 143656e3fdb743fe9a2b30416491f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:48.855206 2574 consensus_queue.cc:1035] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.855834 2574 consensus_queue.cc:1035] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:48.858620 2057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.858749 2057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.859493 2057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Sending DeleteTablet for 3 replicas of tablet e3dec30dba5d4fb39b4c3895acb12d7f
I20260619 02:49:48.859777 2335 tablet_service.cc:1514] Processing DeleteTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260619 02:49:48.859815 2195 tablet_service.cc:1514] Processing DeleteTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:39876
I20260619 02:49:48.859999 2593 tablet_replica.cc:330] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: stopping tablet replica
I20260619 02:49:48.860097 2593 raft_consensus.cc:2229] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.860167 2593 raft_consensus.cc:2258] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.860209 2591 tablet_replica.cc:330] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: stopping tablet replica
I20260619 02:49:48.860277 2591 raft_consensus.cc:2229] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.860342 2591 raft_consensus.cc:2258] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.860562 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet e3dec30dba5d4fb39b4c3895acb12d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:35642
I20260619 02:49:48.860659 2592 tablet_replica.cc:330] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: stopping tablet replica
I20260619 02:49:48.860585 2057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:49:48.860713 2592 raft_consensus.cc:2229] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:48.860785 2592 raft_consensus.cc:2258] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:49:48.860786 2057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:48.861079 2592 ts_tablet_manager.cc:1905] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.861069 2591 ts_tablet_manager.cc:1905] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.863263 2593 ts_tablet_manager.cc:1905] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.863438 2335 tablet_service.cc:1467] Processing CreateTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f (DEFAULT_TABLE table=test-table [id=49a1b8a34cd2411aaa558b75deb5058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.863574 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af5769d2ae14813b52c1a82c7790e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.864741 2571 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Bootstrap starting.
I20260619 02:49:48.864816 2593 ts_tablet_manager.cc:1918] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.864856 2593 log.cc:1196] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/e3dec30dba5d4fb39b4c3895acb12d7f
I20260619 02:49:48.865060 2593 ts_tablet_manager.cc:1939] T e3dec30dba5d4fb39b4c3895acb12d7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting consensus metadata
I20260619 02:49:48.865149 2571 tablet_bootstrap.cc:654] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.865272 2592 ts_tablet_manager.cc:1918] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.865330 2592 log.cc:1196] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/e3dec30dba5d4fb39b4c3895acb12d7f
I20260619 02:49:48.865458 2195 tablet_service.cc:1467] Processing CreateTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f (DEFAULT_TABLE table=test-table [id=49a1b8a34cd2411aaa558b75deb5058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.865511 2592 ts_tablet_manager.cc:1939] T e3dec30dba5d4fb39b4c3895acb12d7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting consensus metadata
I20260619 02:49:48.865844 2475 tablet_service.cc:1467] Processing CreateTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f (DEFAULT_TABLE table=test-table [id=49a1b8a34cd2411aaa558b75deb5058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:48.865882 2045 catalog_manager.cc:4841] TS 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465): tablet e3dec30dba5d4fb39b4c3895acb12d7f (table test-table [id=398f5b77348345f28449cf77e8eb6ddf]) successfully deleted
I20260619 02:49:48.865944 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af5769d2ae14813b52c1a82c7790e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.865948 2571 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: No bootstrap required, opened a new log
I20260619 02:49:48.866004 2571 ts_tablet_manager.cc:1397] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.866151 2571 raft_consensus.cc:348] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.866221 2571 raft_consensus.cc:374] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.866240 2571 raft_consensus.cc:729] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7e90e6f894ac4919d1811ee64b19d, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.866283 2571 consensus_queue.cc:260] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.866382 2571 ts_tablet_manager.cc:1428] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.867069 2569 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Bootstrap starting.
I20260619 02:49:48.867266 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af5769d2ae14813b52c1a82c7790e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:48.867447 2569 tablet_bootstrap.cc:654] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.867558 2042 catalog_manager.cc:4841] TS 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591): tablet e3dec30dba5d4fb39b4c3895acb12d7f (table test-table [id=398f5b77348345f28449cf77e8eb6ddf]) successfully deleted
I20260619 02:49:48.867982 2569 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: No bootstrap required, opened a new log
I20260619 02:49:48.868026 2569 ts_tablet_manager.cc:1397] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.868141 2569 raft_consensus.cc:348] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.868203 2569 raft_consensus.cc:374] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.868223 2569 raft_consensus.cc:729] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143656e3fdb743fe9a2b30416491f0c5, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.868273 2569 consensus_queue.cc:260] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.868382 2569 ts_tablet_manager.cc:1428] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:48.868508 2572 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: Bootstrap starting.
I20260619 02:49:48.867273 2591 ts_tablet_manager.cc:1918] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:48.868832 2591 log.cc:1196] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/e3dec30dba5d4fb39b4c3895acb12d7f
I20260619 02:49:48.868873 2572 tablet_bootstrap.cc:654] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:48.869063 2591 ts_tablet_manager.cc:1939] T e3dec30dba5d4fb39b4c3895acb12d7f P 1438113cbca148818c18b95f53056321: Deleting consensus metadata
I20260619 02:49:48.869377 2044 catalog_manager.cc:4841] TS 1438113cbca148818c18b95f53056321 (127.26.98.1:32961): tablet e3dec30dba5d4fb39b4c3895acb12d7f (table test-table [id=398f5b77348345f28449cf77e8eb6ddf]) successfully deleted
I20260619 02:49:48.869448 2572 tablet_bootstrap.cc:492] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: No bootstrap required, opened a new log
I20260619 02:49:48.869498 2572 ts_tablet_manager.cc:1397] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:48.869624 2572 raft_consensus.cc:348] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.869686 2572 raft_consensus.cc:374] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:48.869699 2572 raft_consensus.cc:729] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1438113cbca148818c18b95f53056321, State: Initialized, Role: FOLLOWER
I20260619 02:49:48.869747 2572 consensus_queue.cc:260] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:48.869846 2572 ts_tablet_manager.cc:1428] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.020530 2625 raft_consensus.cc:482] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:49.020678 2625 raft_consensus.cc:504] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:49.020895 2625 leader_election.cc:288] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:49.021186 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af5769d2ae14813b52c1a82c7790e7f" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d" is_pre_election: true
I20260619 02:49:49.021181 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af5769d2ae14813b52c1a82c7790e7f" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5" is_pre_election: true
I20260619 02:49:49.021286 2495 raft_consensus.cc:2454] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 0.
I20260619 02:49:49.021288 2355 raft_consensus.cc:2454] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 0.
I20260619 02:49:49.021466 2149 leader_election.cc:302] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:49.021564 2625 raft_consensus.cc:2781] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:49.021617 2625 raft_consensus.cc:482] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:49.021633 2625 raft_consensus.cc:3037] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:49.022469 2625 raft_consensus.cc:504] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:49.022611 2625 leader_election.cc:288] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 election: Requested vote from peers 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591), 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465)
I20260619 02:49:49.022789 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af5769d2ae14813b52c1a82c7790e7f" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143656e3fdb743fe9a2b30416491f0c5"
I20260619 02:49:49.022795 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af5769d2ae14813b52c1a82c7790e7f" candidate_uuid: "1438113cbca148818c18b95f53056321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b7e90e6f894ac4919d1811ee64b19d"
I20260619 02:49:49.022862 2495 raft_consensus.cc:3037] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:49.022905 2355 raft_consensus.cc:3037] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:49.023638 2495 raft_consensus.cc:2454] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 1.
I20260619 02:49:49.023712 2355 raft_consensus.cc:2454] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1438113cbca148818c18b95f53056321 in term 1.
I20260619 02:49:49.023829 2149 leader_election.cc:302] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143656e3fdb743fe9a2b30416491f0c5, 1438113cbca148818c18b95f53056321; no voters:
I20260619 02:49:49.023943 2625 raft_consensus.cc:2781] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:49.023996 2625 raft_consensus.cc:686] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 1 LEADER]: Becoming Leader. State: Replica: 1438113cbca148818c18b95f53056321, State: Running, Role: LEADER
I20260619 02:49:49.024056 2625 consensus_queue.cc:237] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } }
I20260619 02:49:49.024712 2057 catalog_manager.cc:5495] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1438113cbca148818c18b95f53056321 (127.26.98.1). New cstate: current_term: 1 leader_uuid: "1438113cbca148818c18b95f53056321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1438113cbca148818c18b95f53056321" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:49.060967 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.058533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.26.98.1:32961)
0619 02:49:49.058694 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.058696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.058703 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:49.058727 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.058728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.058740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.058798 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:49.058959 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.059503 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.060251 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.060470 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.060529 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.060673 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:49.060713 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:49.060756 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.060822 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.060851 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260619 02:49:49.064221 2495 raft_consensus.cc:1261] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Refusing update from remote peer 1438113cbca148818c18b95f53056321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:49.064203 2355 raft_consensus.cc:1261] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Refusing update from remote peer 1438113cbca148818c18b95f53056321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:49.064479 2625 consensus_queue.cc:1035] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "143656e3fdb743fe9a2b30416491f0c5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:49.064675 2599 consensus_queue.cc:1035] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b7e90e6f894ac4919d1811ee64b19d" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:49.075927 2056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:49.076093 2056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:49.077037 2056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae58923d40543d8a1d8138d33252944: Sending DeleteTablet for 3 replicas of tablet 2af5769d2ae14813b52c1a82c7790e7f
I20260619 02:49:49.077437 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:49 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:35642
I20260619 02:49:49.077564 2592 tablet_replica.cc:330] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: stopping tablet replica
I20260619 02:49:49.077529 2195 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:49 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:39876
I20260619 02:49:49.077633 2592 raft_consensus.cc:2229] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:49.077685 2592 raft_consensus.cc:2258] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:49.077666 2591 tablet_replica.cc:330] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321: stopping tablet replica
I20260619 02:49:49.077747 2591 raft_consensus.cc:2229] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:49.077785 2335 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af5769d2ae14813b52c1a82c7790e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:49 UTC) from {username='oryx', principal='oryx/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260619 02:49:49.077841 2591 raft_consensus.cc:2258] T 2af5769d2ae14813b52c1a82c7790e7f P 1438113cbca148818c18b95f53056321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:49.077900 2593 tablet_replica.cc:330] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: stopping tablet replica
I20260619 02:49:49.077977 2593 raft_consensus.cc:2229] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:49.078030 2593 raft_consensus.cc:2258] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:49.078120 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2130
I20260619 02:49:49.078197 2592 ts_tablet_manager.cc:1905] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:49.080063 2593 ts_tablet_manager.cc:1905] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:49.080406 2592 ts_tablet_manager.cc:1918] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:49.080474 2592 log.cc:1196] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/2af5769d2ae14813b52c1a82c7790e7f
I20260619 02:49:49.080734 2592 ts_tablet_manager.cc:1939] T 2af5769d2ae14813b52c1a82c7790e7f P 143656e3fdb743fe9a2b30416491f0c5: Deleting consensus metadata
I20260619 02:49:49.081212 2045 catalog_manager.cc:4841] TS 143656e3fdb743fe9a2b30416491f0c5 (127.26.98.3:39465): tablet 2af5769d2ae14813b52c1a82c7790e7f (table test-table [id=49a1b8a34cd2411aaa558b75deb5058a]) successfully deleted
I20260619 02:49:49.081830 2593 ts_tablet_manager.cc:1918] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:49.081911 2593 log.cc:1196] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/2af5769d2ae14813b52c1a82c7790e7f
I20260619 02:49:49.082121 2593 ts_tablet_manager.cc:1939] T 2af5769d2ae14813b52c1a82c7790e7f P 93b7e90e6f894ac4919d1811ee64b19d: Deleting consensus metadata
I20260619 02:49:49.082641 2042 catalog_manager.cc:4841] TS 93b7e90e6f894ac4919d1811ee64b19d (127.26.98.2:40591): tablet 2af5769d2ae14813b52c1a82c7790e7f (table test-table [id=49a1b8a34cd2411aaa558b75deb5058a]) successfully deleted
W20260619 02:49:49.087021 2044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:32961 (error 108)
I20260619 02:49:49.087298 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2270
I20260619 02:49:49.091212 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.088474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:32961 (local address 127.0.0.1:39902)
0619 02:49:49.091038 (+ 2564us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:49.091145 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2534,"thread_start_us":64,"threads_started":1}
W20260619 02:49:49.091459 2044 catalog_manager.cc:4584] TS 1438113cbca148818c18b95f53056321 (127.26.98.1:32961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2af5769d2ae14813b52c1a82c7790e7f: Network error: Client connection negotiation failed: client connection to 127.26.98.1:32961: connect: Connection refused (error 111)
I20260619 02:49:49.093581 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2410
I20260619 02:49:49.099735 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2026
2026-06-19T02:49:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2984 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:49 dist-test-slave-0f9d krb5kdc[2666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:49 dist-test-slave-0f9d krb5kdc[2666](info): set up 2 sockets
Jun 19 02:49:49 dist-test-slave-0f9d krb5kdc[2666](info): commencing operation
krb5kdc: starting...
W20260619 02:49:50.095947 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:50 dist-test-slave-0f9d krb5kdc[2666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:50.235981 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34489
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:50.335772 2679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:50.336301 2679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:50.336349 2679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:50.338083 2679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:50.338169 2679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:50.338189 2679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:50.338207 2679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:50.338222 2679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:50.340432 2679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:50.340822 2679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:50.341109 2679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:50.347048 2687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:50.347359 2679 server_base.cc:1034] running on GCE node
W20260619 02:49:50.347040 2684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:50.347048 2685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:50.347806 2679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:50.348201 2679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:50.349386 2679 hybrid_clock.cc:648] HybridClock initialized: now 1781837390349324 us; error 82 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:49:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:50 dist-test-slave-0f9d krb5kdc[2692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:50 dist-test-slave-0f9d krb5kdc[2692](info): set up 2 sockets
Jun 19 02:49:50 dist-test-slave-0f9d krb5kdc[2692](info): commencing operation
krb5kdc: starting...
W20260619 02:49:51.351775 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:49:51 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:49:51.490381 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36183
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:51.592000 2705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:51.592301 2705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:51.592330 2705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:51.593756 2705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:51.593816 2705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:51.593830 2705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:51.593842 2705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:51.593853 2705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:51.595818 2705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:51.596148 2705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:51.596396 2705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:51.602480 2711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:51.602480 2710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:51.602480 2713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:51.603025 2705 server_base.cc:1034] running on GCE node
I20260619 02:49:51.603391 2705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:51.603763 2705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:51.604966 2705 hybrid_clock.cc:648] HybridClock initialized: now 1781837391604966 us; error 73 us; skew 500 ppm
Jun 19 02:49:51 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:51.613392 2705 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:51.614050 2705 webserver.cc:460] Webserver started at http://127.26.98.62:46585/ using document root <none> and password file <none>
I20260619 02:49:51.614447 2705 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:51.614511 2705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:51.614670 2705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:51.616061 2705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "22f06db5f8e24e18b5a40cb5ea3fe430"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "2e23cd3ef29af9624df40ec1a7c47350"
server_key_iv: "ee005a9e83b8932d442a322302f8cd05"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.616523 2705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "22f06db5f8e24e18b5a40cb5ea3fe430"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "2e23cd3ef29af9624df40ec1a7c47350"
server_key_iv: "ee005a9e83b8932d442a322302f8cd05"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.618137 2705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:51.619067 2720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:51.619287 2705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:51.619391 2705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "22f06db5f8e24e18b5a40cb5ea3fe430"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "2e23cd3ef29af9624df40ec1a7c47350"
server_key_iv: "ee005a9e83b8932d442a322302f8cd05"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.619522 2705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:51.679703 2705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:51.682315 2705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:51.682488 2705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:51.687870 2705 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36183
I20260619 02:49:51.687920 2772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36183 every 8 connection(s)
I20260619 02:49:51.688349 2705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:51.688607 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2705
I20260619 02:49:51.688723 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:49:51.688948 27016 external_mini_cluster.cc:1428] Setting key 0409e714d8b0d34867de24eb8dee597a
I20260619 02:49:51.689561 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:51.693446 2773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Bootstrap starting.
I20260619 02:49:51.694530 2773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:51.694948 2773 log.cc:824] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:51.696110 2773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: No bootstrap required, opened a new log
Jun 19 02:49:51 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:51.699275 2773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:51.699522 2773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:51.699569 2773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f06db5f8e24e18b5a40cb5ea3fe430, State: Initialized, Role: FOLLOWER
I20260619 02:49:51.699779 2773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:51.699879 2773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:51.699926 2773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:51.699977 2773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:51.700882 2773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:51.701020 2773 leader_election.cc:302] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f06db5f8e24e18b5a40cb5ea3fe430; no voters:
I20260619 02:49:51.701246 2773 leader_election.cc:288] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:51.701295 2778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:51.701460 2778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 LEADER]: Becoming Leader. State: Replica: 22f06db5f8e24e18b5a40cb5ea3fe430, State: Running, Role: LEADER
I20260619 02:49:51.701648 2773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:51.701632 2778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:51.702337 2779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } } }
I20260619 02:49:51.702616 2779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:51.702872 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.689611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.26.98.62:36183)
0619 02:49:51.689855 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.689866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.689902 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:49:51.690506 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.690517 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.690608 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.690805 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.691592 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.693048 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.694260 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.694990 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.695182 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:51.698345 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.698354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.698355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:51.698376 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.700479 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.700954 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.700960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.700961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.701087 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.701288 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.701289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.701290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.701706 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.702382 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.702465 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.702632 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":127,"threads_started":1}
I20260619 02:49:51.702549 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f06db5f8e24e18b5a40cb5ea3fe430. Latest consensus state: current_term: 1 leader_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } } }
I20260619 02:49:51.703102 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:51.703867 2794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:51.703977 2794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:51.704044 2787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:51.704283 2787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:51.706753 2787 catalog_manager.cc:1347] Generated new cluster ID: 06ef92cef52047aa8bac88a18776beeb
I20260619 02:49:51.706837 2787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:51.720407 2787 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:51.721103 2787 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:51.725348 2787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Generated new TSK 0
I20260619 02:49:51.725654 2787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:49:51.770064 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:51.892688 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:51.893003 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:51.893044 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:51.894531 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:51.894601 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:51.894644 2799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:51.896497 2799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:51.896821 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:51.897087 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:51.902802 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:51.902834 2799 server_base.cc:1034] running on GCE node
W20260619 02:49:51.902945 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:51.902801 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:51.903290 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:51.903591 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:51.904776 2799 hybrid_clock.cc:648] HybridClock initialized: now 1781837391904759 us; error 39 us; skew 500 ppm
Jun 19 02:49:51 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:51.908420 2799 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:49:51.909087 2799 webserver.cc:460] Webserver started at http://127.26.98.1:34607/ using document root <none> and password file <none>
I20260619 02:49:51.909382 2799 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:51.909442 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:51.909600 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:51.911033 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "3e469763573d44a4b61605659fef9d3e"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "1c4b6bac5df013b210ba4de54e6ced11"
server_key_iv: "2583142c1b7613a02c61ad961f9cc5b1"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.911521 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "3e469763573d44a4b61605659fef9d3e"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "1c4b6bac5df013b210ba4de54e6ced11"
server_key_iv: "2583142c1b7613a02c61ad961f9cc5b1"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.913200 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:51.914456 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:51.914669 2799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:51.914770 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3e469763573d44a4b61605659fef9d3e"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "1c4b6bac5df013b210ba4de54e6ced11"
server_key_iv: "2583142c1b7613a02c61ad961f9cc5b1"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.914865 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:51.929840 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:51.932346 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:51.932515 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:51.932801 2799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:51.933149 2799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:51.933187 2799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:51.933223 2799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:51.933238 2799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:51.939941 2799 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:40769
I20260619 02:49:51.939988 2927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:40769 every 8 connection(s)
I20260619 02:49:51.940383 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:51.947840 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2799
I20260619 02:49:51.948169 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:51.948432 27016 external_mini_cluster.cc:1428] Setting key 3661418677da39983a9067cf6446c73b
Jun 19 02:49:51 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:51.953658 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.941475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:38273 (local address 127.26.98.62:36183)
0619 02:49:51.941539 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.941541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.941557 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:49:51.942335 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.942337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.942350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.942423 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.943869 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.944554 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.945867 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.946095 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.946168 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:51.949188 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.949191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.949191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:51.949203 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.951085 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.951756 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.951758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.951759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.951853 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.952720 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.952722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.952722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.952916 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.953002 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.953402 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.953537 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:51.953923 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.941239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:38273)
0619 02:49:51.941426 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:51.941478 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:51.942063 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:51.942666 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:51.942671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:51.943422 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:51.943700 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:51.943707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.944749 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.944750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:51.945717 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:51.945722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.946186 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.946187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:51.946323 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:51.947018 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:51.947028 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:51.949037 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:51.951264 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:51.951283 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:51.951287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:51.951641 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:51.951961 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:51.951963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:51.951963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:51.952190 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:51.953003 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:51.953020 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:51.953243 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 02:49:51.953359 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:49:51.953631 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260619 02:49:51.954540 2928 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36183
I20260619 02:49:51.954658 2928 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:51.954964 2928 heartbeater.cc:511] Master 127.26.98.62:36183 requested a full tablet report, sending...
I20260619 02:49:51.955685 2737 ts_manager.cc:194] Registered new tserver with Master: 3e469763573d44a4b61605659fef9d3e (127.26.98.1:40769)
I20260619 02:49:51.956382 2737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:38273
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:49:51.978641 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.076885 2933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.077185 2933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.077216 2933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.078743 2933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.078828 2933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.078867 2933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:52.081025 2933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:52.081394 2933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.081664 2933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.087875 2939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.087880 2941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.087935 2938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.088419 2933 server_base.cc:1034] running on GCE node
I20260619 02:49:52.088699 2933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.088990 2933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.090164 2933 hybrid_clock.cc:648] HybridClock initialized: now 1781837392090141 us; error 52 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.093854 2933 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:49:52.094588 2933 webserver.cc:460] Webserver started at http://127.26.98.2:41363/ using document root <none> and password file <none>
I20260619 02:49:52.094946 2933 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.095022 2933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.095196 2933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.096557 2933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "e77a1c60fb8144f29c7c3d5731937004"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "8b960c24b62270f9a7d4536453be4e43"
server_key_iv: "ad015b2d1a00af55155f2763cf8e1e2b"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.096997 2933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "e77a1c60fb8144f29c7c3d5731937004"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "8b960c24b62270f9a7d4536453be4e43"
server_key_iv: "ad015b2d1a00af55155f2763cf8e1e2b"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.099020 2933 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:52.100389 2948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.100745 2933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:52.100874 2933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "e77a1c60fb8144f29c7c3d5731937004"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "8b960c24b62270f9a7d4536453be4e43"
server_key_iv: "ad015b2d1a00af55155f2763cf8e1e2b"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.100981 2933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.136435 2933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.139479 2933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.139678 2933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.139961 2933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.140328 2933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.140368 2933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.140403 2933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.140426 2933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.147539 2933 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:41293
I20260619 02:49:52.147670 3061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:41293 every 8 connection(s)
I20260619 02:49:52.148034 2933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:52.156869 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 2933
I20260619 02:49:52.156981 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:52.157205 27016 external_mini_cluster.cc:1428] Setting key a1bc260e9c085ad38dfe794e79946469
I20260619 02:49:52.160790 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.149151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:51789 (local address 127.26.98.62:36183)
0619 02:49:52.149247 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.149249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.149466 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:49:52.150072 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.150073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.150091 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.150167 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.151578 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.152355 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.154180 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.154399 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.154463 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.157255 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.157258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.157259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.157271 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.159044 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.159529 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.159530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.159532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.159606 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.159876 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.159877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.159878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.160089 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.160214 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.160554 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.160676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:49:52.161023 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.149103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:51789)
0619 02:49:52.149287 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.149342 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.149932 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.150274 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.150278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.151062 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.151349 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.151356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.152541 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.152543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.154037 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.154041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.154494 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.154495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.154643 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.155280 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.155292 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.157085 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:52.159135 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.159146 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.159149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.159447 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.159677 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.159677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.159678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.159804 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.160217 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:52.160226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:52.160416 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:49:52.160527 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:49:52.160747 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260619 02:49:52.161720 3062 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36183
I20260619 02:49:52.161957 3062 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:52.162222 3062 heartbeater.cc:511] Master 127.26.98.62:36183 requested a full tablet report, sending...
I20260619 02:49:52.162743 2737 ts_manager.cc:194] Registered new tserver with Master: e77a1c60fb8144f29c7c3d5731937004 (127.26.98.2:41293)
I20260619 02:49:52.163208 2737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:51789
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:49:52.188088 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.285531 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.285840 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.285871 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.287767 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.287856 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.287896 3067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:52.289884 3067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:52.290205 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.290455 3067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.296082 3073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.296248 3072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.296391 3075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.296836 3067 server_base.cc:1034] running on GCE node
I20260619 02:49:52.297124 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.297427 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.298586 3067 hybrid_clock.cc:648] HybridClock initialized: now 1781837392298517 us; error 116 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.301635 3067 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:49:52.302306 3067 webserver.cc:460] Webserver started at http://127.26.98.3:39727/ using document root <none> and password file <none>
I20260619 02:49:52.302587 3067 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.302645 3067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.302835 3067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.304349 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "effd81ebfb7f4707bf1d2f968992288f"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "da4a99ac62eeac35cfd0e5200fdfa06d"
server_key_iv: "21939020788d72b0197dabb90ed96df1"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.304800 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "effd81ebfb7f4707bf1d2f968992288f"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "da4a99ac62eeac35cfd0e5200fdfa06d"
server_key_iv: "21939020788d72b0197dabb90ed96df1"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.306443 3067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 02:49:52.307385 3082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.307626 3067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:52.307749 3067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "effd81ebfb7f4707bf1d2f968992288f"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "da4a99ac62eeac35cfd0e5200fdfa06d"
server_key_iv: "21939020788d72b0197dabb90ed96df1"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.307835 3067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.330871 3067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.333379 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.333551 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.333840 3067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.334190 3067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.334225 3067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.334268 3067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.334292 3067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.341280 3067 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:41555
I20260619 02:49:52.341323 3195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:41555 every 8 connection(s)
I20260619 02:49:52.341718 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:52.345103 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3067
I20260619 02:49:52.345198 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:52.345371 27016 external_mini_cluster.cc:1428] Setting key f060b38648c4861fe5facf0a25f58a47
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:49:52.354260 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.342629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:46499 (local address 127.26.98.62:36183)
0619 02:49:52.342692 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.342694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.342859 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:49:52.343543 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.343544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.343557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.343639 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.344779 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.345387 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.346996 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.347233 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.347303 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.350651 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.350654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.350654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.350665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.352437 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.353131 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.353133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.353134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.353207 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.353507 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.353508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.353509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.353669 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.353735 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.354078 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.354176 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:49:52.354447 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.342539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:46499)
0619 02:49:52.342725 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.342779 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.343399 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.343746 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.343750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.344401 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.344657 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.344662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.345504 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.345506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.346841 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.346844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.347331 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.347332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.347483 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.348275 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.348290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.350491 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:52.352590 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.352605 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.352610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.352985 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.353276 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.353276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.353277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.353419 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.353743 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:52.353750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:52.353924 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 02:49:52.354034 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:49:52.354209 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260619 02:49:52.354988 3196 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36183
I20260619 02:49:52.355116 3196 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:52.355329 3196 heartbeater.cc:511] Master 127.26.98.62:36183 requested a full tablet report, sending...
I20260619 02:49:52.355764 2737 ts_manager.cc:194] Registered new tserver with Master: effd81ebfb7f4707bf1d2f968992288f (127.26.98.3:41555)
I20260619 02:49:52.356168 2737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:46499
I20260619 02:49:52.357568 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
Entry for principal oryx/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab.
WARNING: no policy specified for oryx/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
Entry for principal oryx/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab.
WARNING: no policy specified for oryx/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
Entry for principal oryx/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab.
I20260619 02:49:52.447257 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2799
I20260619 02:49:52.452513 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2933
I20260619 02:49:52.457710 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3067
I20260619 02:49:52.462728 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 2705
I20260619 02:49:52.468771 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36183
--webserver_interface=127.26.98.62
--webserver_port=46585
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.568146 3205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.568492 3205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.568547 3205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.570199 3205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:52.570286 3205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.570309 3205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.570330 3205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:52.570348 3205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:52.572443 3205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=46585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:52.572825 3205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.573108 3205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.578496 3211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.578630 3210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.578589 3213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.578873 3205 server_base.cc:1034] running on GCE node
I20260619 02:49:52.579187 3205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.579597 3205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.580809 3205 hybrid_clock.cc:648] HybridClock initialized: now 1781837392580782 us; error 39 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.584273 3205 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:49:52.584854 3205 webserver.cc:460] Webserver started at http://127.26.98.62:46585/ using document root <none> and password file <none>
I20260619 02:49:52.585244 3205 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.585330 3205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.587239 3205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:52.588435 3220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.588806 3205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:52.588939 3205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "22f06db5f8e24e18b5a40cb5ea3fe430"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "2e23cd3ef29af9624df40ec1a7c47350"
server_key_iv: "ee005a9e83b8932d442a322302f8cd05"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.589527 3205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.604724 3205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.607540 3205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.607726 3205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.612869 3205 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36183
I20260619 02:49:52.612918 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36183 every 8 connection(s)
I20260619 02:49:52.613557 3205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:52.615538 3273 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:49:52.615689 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3205
I20260619 02:49:52.616030 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:40769
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:36183
--webserver_port=34607
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:49:52.616611 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Bootstrap starting.
I20260619 02:49:52.630478 3273 log.cc:824] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:52.633484 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:49:52.633932 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Bootstrap complete.
I20260619 02:49:52.637394 3273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:52.637830 3273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f06db5f8e24e18b5a40cb5ea3fe430, State: Initialized, Role: FOLLOWER
I20260619 02:49:52.638031 3273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:52.638105 3273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:52.638156 3273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:52.638209 3273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:49:52.640179 3273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:52.640383 3273 leader_election.cc:302] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f06db5f8e24e18b5a40cb5ea3fe430; no voters:
I20260619 02:49:52.640658 3273 leader_election.cc:288] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:49:52.640796 3277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:49:52.640964 3277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [term 2 LEADER]: Becoming Leader. State: Replica: 22f06db5f8e24e18b5a40cb5ea3fe430, State: Running, Role: LEADER
I20260619 02:49:52.640985 3273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:52.641076 3277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } }
I20260619 02:49:52.641492 3278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } } }
I20260619 02:49:52.641587 3278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:52.641966 3285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:52.642434 3279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f06db5f8e24e18b5a40cb5ea3fe430. Latest consensus state: current_term: 2 leader_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f06db5f8e24e18b5a40cb5ea3fe430" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36183 } } }
I20260619 02:49:52.642612 3279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:52.643201 3285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:52.643339 3285 catalog_manager.cc:1259] Loaded cluster ID: 06ef92cef52047aa8bac88a18776beeb
I20260619 02:49:52.643374 3285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:52.644743 3285 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:52.644996 3285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22f06db5f8e24e18b5a40cb5ea3fe430: Loaded TSK: 0
I20260619 02:49:52.645257 3285 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:49:52.721853 3275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.722193 3275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.722234 3275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.723979 3275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.724062 3275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.724117 3275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:49:52.726078 3275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.1.keytab
--principal=oryx/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:40769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=34607
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:52.726440 3275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.726720 3275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.732728 3302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.732743 3299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.732728 3300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.733667 3275 server_base.cc:1034] running on GCE node
I20260619 02:49:52.734026 3275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.734409 3275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.735616 3275 hybrid_clock.cc:648] HybridClock initialized: now 1781837392735586 us; error 51 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.739583 3275 init.cc:377] Logged in from keytab as oryx/127.26.98.1@KRBTEST.COM (short username oryx)
I20260619 02:49:52.740206 3275 webserver.cc:460] Webserver started at http://127.26.98.1:34607/ using document root <none> and password file <none>
I20260619 02:49:52.740504 3275 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.740594 3275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.742537 3275 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:52.743425 3309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.743623 3275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:52.743727 3275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3e469763573d44a4b61605659fef9d3e"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-0f9d"
server_key: "1c4b6bac5df013b210ba4de54e6ced11"
server_key_iv: "2583142c1b7613a02c61ad961f9cc5b1"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.744215 3275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.776190 3275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.778815 3275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.779032 3275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.779340 3275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.779716 3275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.779758 3275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.779793 3275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.779815 3275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.787510 3275 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:40769
I20260619 02:49:52.787565 3422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:40769 every 8 connection(s)
I20260619 02:49:52.787974 3275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:52.794137 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3275
I20260619 02:49:52.794472 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:41293
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:36183
--webserver_port=41363
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:52.798051 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.788801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:58983)
0619 02:49:52.788998 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.789044 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.789702 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.790186 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.790191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.790909 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.791159 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.791166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.792441 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.792442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.793455 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.793459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.794120 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.794122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.794272 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.794964 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.794975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.797837 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
W20260619 02:49:52.798362 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.788949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:58983 (local address 127.26.98.62:36183)
0619 02:49:52.789182 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.789191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.789246 (+ 55us) server_negotiation.cc:407] Connection header received
0619 02:49:52.789934 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.789939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.789990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.790123 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.791327 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.792305 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.793553 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.794018 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.794142 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.798120 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:58983: BlockingRecv error: recv got EOF from 127.26.98.1:58983 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
W20260619 02:49:52.798677 3423 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36183 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.98.62:36183: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:52.806354 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.799196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:60961)
0619 02:49:52.799286 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.799302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.799377 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.799601 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.799603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.799815 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.799907 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.799911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.800793 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.800794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.801758 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.801764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.802433 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.802435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.802520 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.803249 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.803257 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.806241 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 02:49:52.806845 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.799367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:60961 (local address 127.26.98.62:36183)
0619 02:49:52.799427 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.799429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.799439 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:52.799470 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.799471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.799487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.799551 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.800086 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.800646 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.802004 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.802290 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.802392 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.806408 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:60961: BlockingRecv error: recv got EOF from 127.26.98.1:60961 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:52.812680 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.806732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:57321)
0619 02:49:52.806796 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.806812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.806905 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.807148 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.807150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.807332 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.807394 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.807397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.808405 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.808406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.809371 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.809375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.809816 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.809817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.809881 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.810406 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.810411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.812578 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:49:52.813071 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.806800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:57321 (local address 127.26.98.62:36183)
0619 02:49:52.806966 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.806968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.806977 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:52.807011 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.807012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.807067 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.807129 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.807555 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.808261 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.809497 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.809730 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.809808 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.812979 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:57321: BlockingRecv error: recv got EOF from 127.26.98.1:57321 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":193,"threads_started":1}
W20260619 02:49:52.813024 3423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:49:52.896387 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.896694 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.896735 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.898322 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.898406 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.898452 3427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:49:52.900724 3427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.2.keytab
--principal=oryx/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=41363
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:52.901099 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.901373 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.907208 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.907198 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.907196 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.907393 3427 server_base.cc:1034] running on GCE node
I20260619 02:49:52.907766 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.908058 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.909240 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781837392909216 us; error 37 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.912578 3427 init.cc:377] Logged in from keytab as oryx/127.26.98.2@KRBTEST.COM (short username oryx)
I20260619 02:49:52.913406 3427 webserver.cc:460] Webserver started at http://127.26.98.2:41363/ using document root <none> and password file <none>
I20260619 02:49:52.913689 3427 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.913758 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.915416 3427 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:52.916165 3443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.916392 3427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:52.916487 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "e77a1c60fb8144f29c7c3d5731937004"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "8b960c24b62270f9a7d4536453be4e43"
server_key_iv: "ad015b2d1a00af55155f2763cf8e1e2b"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.916954 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.928802 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.932097 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.932263 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.932507 3427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.932858 3427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.932899 3427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.932924 3427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.932936 3427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.940464 3427 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:41293
I20260619 02:49:52.940558 3556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:41293 every 8 connection(s)
I20260619 02:49:52.941648 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:49:52.950932 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.942665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:37715 (local address 127.26.98.62:36183)
0619 02:49:52.942766 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.942771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.943023 (+ 252us) server_negotiation.cc:407] Connection header received
0619 02:49:52.943690 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.943692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.943713 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.943784 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.945020 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.945696 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.947098 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.947322 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.947386 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.950825 (+ 3439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:37715: BlockingRecv error: recv got EOF from 127.26.98.2:37715 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:49:52.950992 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.942612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:37715)
0619 02:49:52.942843 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.942931 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.943547 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.943908 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.943913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.944631 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.944889 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.944894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.945912 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.945913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.946992 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.946996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.947436 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.947438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.947606 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.948278 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.948288 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.950702 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
W20260619 02:49:52.951242 3557 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36183 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.98.62:36183: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
I20260619 02:49:52.951681 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3427
I20260619 02:49:52.951994 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:41555
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:36183
--webserver_port=39727
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:52.956079 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.951478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:48473)
0619 02:49:52.951524 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.951535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.951584 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.951730 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.951731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.951894 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.951957 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.951959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.952680 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.952681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.953240 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.953242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.953585 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.953586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.953651 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.954172 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.954176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.955995 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:49:52.956383 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.951552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:48473 (local address 127.26.98.62:36183)
0619 02:49:52.951624 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.951626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.951635 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:52.951664 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.951665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.951675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.951718 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.952029 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.952605 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.953326 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.953518 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.953574 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.956109 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:48473: BlockingRecv error: recv got EOF from 127.26.98.2:48473 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:52 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:52.961539 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.956341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:53623)
0619 02:49:52.956397 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.956410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.956465 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.956601 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.956602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.956776 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.956831 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.956833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.957536 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.957537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.958337 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.958340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.958753 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.958754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.958815 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.959315 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.959322 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.961448 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:49:52.961714 3557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:49:52.961819 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:52.956413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:53623 (local address 127.26.98.62:36183)
0619 02:49:52.956483 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.956485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.956495 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:52.956521 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.956522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.956532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.956586 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.956927 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.957453 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.958448 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.958657 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.958732 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.961569 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:53623: BlockingRecv error: recv got EOF from 127.26.98.2:53623 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 02:49:53.054735 3560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:53.055140 3560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:53.055178 3560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:53.056758 3560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:53.056833 3560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:53.056869 3560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:49:53.059401 3560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/krb5kdc/oryx_127.26.98.3.keytab
--principal=oryx/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=39727
--tserver_master_addrs=127.26.98.62:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:49:53.059789 3560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:53.060087 3560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:53.066661 3566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:53.066671 3568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:53.066843 3560 server_base.cc:1034] running on GCE node
W20260619 02:49:53.066648 3565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:53.067292 3560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:53.067633 3560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:53.068812 3560 hybrid_clock.cc:648] HybridClock initialized: now 1781837393068803 us; error 39 us; skew 500 ppm
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837393, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:53.071834 3560 init.cc:377] Logged in from keytab as oryx/127.26.98.3@KRBTEST.COM (short username oryx)
I20260619 02:49:53.072395 3560 webserver.cc:460] Webserver started at http://127.26.98.3:39727/ using document root <none> and password file <none>
I20260619 02:49:53.072657 3560 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:53.072726 3560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:53.074535 3560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:53.075609 3575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:53.076064 3560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:53.076195 3560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "effd81ebfb7f4707bf1d2f968992288f"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-0f9d"
server_key: "da4a99ac62eeac35cfd0e5200fdfa06d"
server_key_iv: "21939020788d72b0197dabb90ed96df1"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.076699 3560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:53.091849 3560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:53.094560 3560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:53.094743 3560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:53.095079 3560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:53.095484 3560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:53.095523 3560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:53.095577 3560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:53.095602 3560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:53.105636 3560 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:41555
I20260619 02:49:53.105683 3688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:41555 every 8 connection(s)
I20260619 02:49:53.106165 3560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:53.109054 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3560
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:49:53.117000 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:53.107472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:48465 (local address 127.26.98.62:36183)
0619 02:49:53.107574 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.107576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.107934 (+ 358us) server_negotiation.cc:407] Connection header received
0619 02:49:53.108747 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.108749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.108767 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.108860 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.110429 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.111038 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.112299 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.112509 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.112569 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.116929 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:48465: BlockingRecv error: recv got EOF from 127.26.98.3:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:49:53.117101 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.107429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:48465)
0619 02:49:53.107639 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:53.107694 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:53.108478 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:53.109160 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:53.109167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:53.110021 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:53.110315 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.110322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.111158 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.111159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.112194 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.112198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.112604 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.112605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.112773 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.113378 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:53.113388 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:53.116817 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
W20260619 02:49:53.117971 3689 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36183 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.98.62:36183: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
I20260619 02:49:53.118793 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.109904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.26.98.62:36183)
0619 02:49:53.109997 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.109999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.110009 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:53.110041 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.110042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.110056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.110128 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.110791 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.111381 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.112245 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.112482 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.112548 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.114288 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:53.114296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:53.114296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:53.114311 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:53.116602 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.117061 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.117067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.117067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.117224 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.117490 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.117491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.117491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.117899 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:53.118092 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:53.118490 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 02:49:53.118716 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:53.124325 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.118258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:38323)
0619 02:49:53.118338 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:53.118351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:53.118407 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:53.118535 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:53.118536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:53.118683 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:53.118767 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.118769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.119706 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.119709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.120453 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.120457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.121082 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.121083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.121197 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.121704 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:53.121709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:53.124194 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:49:53.124403 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:53.118364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:38323 (local address 127.26.98.62:36183)
0619 02:49:53.118416 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.118417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.118428 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:53.118452 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.118453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.118464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.118530 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.118837 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.119582 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.120666 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.120926 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.121058 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.124340 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:38323: BlockingRecv error: recv got EOF from 127.26.98.3:38323 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:53.129308 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.124727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:49469)
0619 02:49:53.124777 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:53.124789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:53.124830 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:53.125018 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:53.125019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:53.125156 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:53.125201 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.125203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.125883 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.125885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.126417 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.126419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.126761 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.126762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.126814 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.127348 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:53.127353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:53.129239 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 02:49:53.129594 3426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:53.124852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:49469 (local address 127.26.98.62:36183)
0619 02:49:53.124916 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.124918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.124927 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:53.124952 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.124952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.124962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.125007 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.125267 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.125809 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.126494 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.126699 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.126764 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.129520 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:49469: BlockingRecv error: recv got EOF from 127.26.98.3:49469 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 02:49:53.129587 3689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:53.819749 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.813669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:46509)
0619 02:49:53.813858 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:53.813872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:53.813956 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:53.814206 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:53.814207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:53.814414 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:53.814512 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.814515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.815447 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.815448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.816082 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.816085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.816609 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.816610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.816697 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.817331 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:53.817336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:53.819661 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260619 02:49:53.819914 3694 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:53.813823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:46509 (local address 127.26.98.62:36183)
0619 02:49:53.813955 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.813957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.813988 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:53.814024 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.814024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.814037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.814196 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.814679 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.815329 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.816186 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.816420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.816494 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.819830 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:46509: BlockingRecv error: recv got EOF from 127.26.98.1:46509 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:53 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:53.968039 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.962316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:42569)
0619 02:49:53.962470 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:53.962493 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:53.962588 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:53.962770 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:53.962771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:53.963003 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:53.963067 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.963069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.963846 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.963847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.964475 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:53.964477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.964930 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.964931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:53.964991 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.965654 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:53.965660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:53.967945 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260619 02:49:53.968199 3694 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:53.962454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:42569 (local address 127.26.98.62:36183)
0619 02:49:53.962558 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.962560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.962571 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:53.962646 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.962647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.962662 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.962715 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.963176 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.963756 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.964603 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.964830 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.964897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.968098 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:42569: BlockingRecv error: recv got EOF from 127.26.98.2:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:54.137274 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.130271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:55497)
0619 02:49:54.130460 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.130472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.130568 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.130837 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.130839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.131085 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.131159 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.131163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.132076 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.132077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.133076 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.133082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.133571 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.133572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.133626 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.134514 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.134520 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.137163 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
W20260619 02:49:54.137439 3694 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:54.130390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:55497 (local address 127.26.98.62:36183)
0619 02:49:54.130448 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.130451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.130613 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:49:54.130670 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.130671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.130685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.130822 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.131274 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.131874 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.133211 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.133479 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.133566 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.137355 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:55497: BlockingRecv error: recv got EOF from 127.26.98.3:55497 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:54.826701 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.820451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:59021)
0619 02:49:54.820662 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.820675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.820742 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.820999 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.821001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.821267 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.821345 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.821348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.822124 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.822126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.822804 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.822809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.823441 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.823442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.823507 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.824145 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.824151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.826614 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
W20260619 02:49:54.826853 3698 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:54.820636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:59021 (local address 127.26.98.62:36183)
0619 02:49:54.820769 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.820771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.820780 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:54.820808 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.820809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.820824 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.820930 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.821430 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.822000 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.823065 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.823317 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.823401 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.826773 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:59021: BlockingRecv error: recv got EOF from 127.26.98.1:59021 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:54 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:54.974133 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.968826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:48511)
0619 02:49:54.969024 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.969037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.969125 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.969287 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.969289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.969480 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.969545 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.969548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.970254 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.970255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.971005 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.971008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.971385 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.971385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.971439 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.971912 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.971916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.974008 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
W20260619 02:49:54.974224 3698 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:54.968956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:48511 (local address 127.26.98.62:36183)
0619 02:49:54.969020 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.969022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.969112 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:49:54.969181 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.969182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.969195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.969247 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.969624 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.970191 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.971119 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.971323 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.971382 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.974163 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:48511: BlockingRecv error: recv got EOF from 127.26.98.2:48511 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:55.144290 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.138115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:59053)
0619 02:49:55.138342 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:55.138364 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:55.138463 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:55.138756 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:55.138758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:55.139035 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:55.139152 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.139156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.139928 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.139929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.140514 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.140518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.140915 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.140916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.140984 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.141611 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:55.141617 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:55.144196 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
W20260619 02:49:55.144538 3698 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:55.138365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59053 (local address 127.26.98.62:36183)
0619 02:49:55.138440 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.138443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.138454 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:55.138538 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.138539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.138622 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.138720 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.139243 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.139817 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.140627 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.140829 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.140896 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.144465 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:59053: BlockingRecv error: recv got EOF from 127.26.98.3:59053 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:55.833124 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.827485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:40617)
0619 02:49:55.827670 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:55.827683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:55.827757 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:55.827937 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:55.827938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:55.828165 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:55.828229 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.828231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.828945 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.828946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.829522 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.829525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.829965 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.829966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.830022 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.830613 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:55.830620 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:55.833044 (+ 2424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
W20260619 02:49:55.833307 3702 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:55.827654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:40617 (local address 127.26.98.62:36183)
0619 02:49:55.827782 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.827784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.827795 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:55.827825 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.827826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.827838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.827905 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.828296 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.828854 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.829639 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.829862 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.829935 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.833227 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:40617: BlockingRecv error: recv got EOF from 127.26.98.1:40617 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:55 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:55.980198 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.974866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:57961)
0619 02:49:55.975179 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:55.975194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:55.975293 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:55.975465 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:55.975466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:55.975682 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:55.975747 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.975749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.976481 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.976483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.977031 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.977033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.977400 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.977400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.977459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.977985 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:55.977991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:55.980109 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":262,"threads_started":1}
W20260619 02:49:55.980343 3702 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:55.975154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:57961 (local address 127.26.98.62:36183)
0619 02:49:55.975233 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.975235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.975267 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:55.975374 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.975375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.975391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.975450 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.975829 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.976392 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.977126 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.977331 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.977394 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.980232 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:57961: BlockingRecv error: recv got EOF from 127.26.98.2:57961 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:56.151412 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.144959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:59249)
0619 02:49:56.145179 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.145198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.145314 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.145635 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.145636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.145844 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.145908 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.145910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.146737 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.146738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.147467 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.147470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.148121 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.148122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.148189 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.148835 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:56.148843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.151320 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
W20260619 02:49:56.151607 3702 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:56.145171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59249 (local address 127.26.98.62:36183)
0619 02:49:56.145295 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.145298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.145315 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:56.145511 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.145513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.145530 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.145608 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.146039 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.146595 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.147606 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.147912 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.148112 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.151518 (+ 3406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:59249: BlockingRecv error: recv got EOF from 127.26.98.3:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:56.843946 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.833958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:37263)
0619 02:49:56.834116 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.834128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.834206 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.834426 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.834427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.834612 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.834676 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.834679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.835511 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.835513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.836122 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.836124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.836538 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.836545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.836609 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.841156 (+ 4547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:56.841192 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.843862 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260619 02:49:56.844120 3706 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:56.834009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:37263 (local address 127.26.98.62:36183)
0619 02:49:56.834187 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.834190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.834201 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:56.834289 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.834289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.834305 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.834385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.834765 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.835392 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.836234 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.836445 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.836501 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.844027 (+ 7526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:37263: BlockingRecv error: recv got EOF from 127.26.98.1:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:56 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:56.986965 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.980943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:39253)
0619 02:49:56.981091 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.981105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.981195 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.981414 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.981415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.981650 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.981716 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.981719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.982715 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.982717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.983460 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.983465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.983947 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.983948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.984005 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.984585 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:56.984590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.986838 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
W20260619 02:49:56.987145 3706 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:56.981071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:39253 (local address 127.26.98.62:36183)
0619 02:49:56.981140 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.981142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.981170 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:49:56.981291 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.981292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.981307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.981374 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.981826 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.982566 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.983611 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.983838 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.983912 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.987058 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:39253: BlockingRecv error: recv got EOF from 127.26.98.2:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:57.159756 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.152253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:41359)
0619 02:49:57.152497 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.152514 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.152624 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.152930 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.152931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.153140 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:57.153210 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.153213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.154035 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.154037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.154852 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.154857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.155635 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.155636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.155705 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.156516 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:57.156523 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:57.159586 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20260619 02:49:57.159905 3706 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.152447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:41359 (local address 127.26.98.62:36183)
0619 02:49:57.152571 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.152574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.152605 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:57.152778 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.152779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.152798 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.152876 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.153350 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.153930 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.155306 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.155555 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.155625 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.159821 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:41359: BlockingRecv error: recv got EOF from 127.26.98.3:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:57.851017 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.844720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:33755)
0619 02:49:57.844991 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.845006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.845103 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.845476 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.845478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.845699 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:57.845762 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.845765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.846647 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.846649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.847359 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.847362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.847832 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.847833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.847893 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.848562 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:57.848568 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:57.850933 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":123,"threads_started":1}
W20260619 02:49:57.851313 3710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.844923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:33755 (local address 127.26.98.62:36183)
0619 02:49:57.845210 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.845213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.845224 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:57.845262 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.845263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.845276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.845364 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.845891 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.846500 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.847473 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.847705 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.847789 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.851222 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:33755: BlockingRecv error: recv got EOF from 127.26.98.1:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":97,"threads_started":1}
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:57 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:57.993031 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.987648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:41893)
0619 02:49:57.987862 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.987875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.987961 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.988155 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.988156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.988351 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:57.988411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.988414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.989140 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.989141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.989821 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.989824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.990285 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.990285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.990354 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.990915 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:57.990921 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:57.992951 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
W20260619 02:49:57.993153 3710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.987854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:41893 (local address 127.26.98.62:36183)
0619 02:49:57.987928 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.987930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.987940 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:57.988034 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.988035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.988050 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.988106 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.988491 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.989044 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.989920 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.990216 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.990278 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.993072 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:41893: BlockingRecv error: recv got EOF from 127.26.98.2:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:58.166553 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:58.160433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:34615)
0619 02:49:58.160598 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:58.160614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:58.160685 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:58.160925 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:58.160926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:58.161158 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:58.161222 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.161224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.162050 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.162051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.162701 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.162705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.163168 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.163169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.163242 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.163810 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:58.163814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:58.166440 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260619 02:49:58.166750 3710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:58.160528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:34615 (local address 127.26.98.62:36183)
0619 02:49:58.160622 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:58.160624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:58.160665 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:58.160790 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:58.160791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:58.160806 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:58.160867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:58.161329 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.161908 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.162816 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.163067 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.163135 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.166676 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:34615: BlockingRecv error: recv got EOF from 127.26.98.3:34615 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:58.857292 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:58.851718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:37735)
0619 02:49:58.851864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:58.851877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:58.851954 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:58.852119 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:58.852120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:58.852325 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:58.852382 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.852385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.853286 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.853288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.853900 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.853903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.854358 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.854359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.854421 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.855121 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:58.855127 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:58.857184 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260619 02:49:58.857385 3714 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:58.851839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:37735 (local address 127.26.98.62:36183)
0619 02:49:58.851966 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:58.851968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:58.851994 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:49:58.852025 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:58.852026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:58.852037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:58.852103 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:58.852481 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.853211 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.854026 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.854259 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.854325 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.857314 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:37735: BlockingRecv error: recv got EOF from 127.26.98.1:37735 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:58 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:58.999377 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:58.993597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:41089)
0619 02:49:58.993762 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:58.993773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:58.993846 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:58.994111 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:58.994113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:58.994312 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:58.994372 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.994374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.995258 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.995259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.995880 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:58.995883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.996333 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.996335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:58.996396 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.997015 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:58.997021 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:58.999283 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
W20260619 02:49:58.999539 3714 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:58.993722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:41089 (local address 127.26.98.62:36183)
0619 02:49:58.993782 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:58.993793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:58.993826 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:58.993938 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:58.993939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:58.993953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:58.994032 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:58.994532 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.995122 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.995996 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:58.996223 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:58.996298 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:58.999459 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:41089: BlockingRecv error: recv got EOF from 127.26.98.2:41089 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 02:49:59 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:59 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:59.173576 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.167249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:48391)
0619 02:49:59.167406 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.167417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.167483 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.167726 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.167727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.167910 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.167975 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.167977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.169035 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.169037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.169693 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.169697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.170141 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.170142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.170215 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.171050 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.171056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.173501 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
W20260619 02:49:59.173786 3714 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:59.167347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:48391 (local address 127.26.98.62:36183)
0619 02:49:59.167482 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.167484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.167538 (+ 54us) server_negotiation.cc:407] Connection header received
0619 02:49:59.167576 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.167578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.167598 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.167662 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.168068 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.168891 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.169825 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.170036 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.170105 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.173710 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:48391: BlockingRecv error: recv got EOF from 127.26.98.3:48391 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 02:49:59 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:49:59 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:49:59.864171 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.858080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:43659)
0619 02:49:59.858323 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.858339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.858434 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.858729 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.858730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.858968 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.859034 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.859036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.859861 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.859862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.860529 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.860532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.860971 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.860971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.861066 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.861670 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.861675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.864041 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
W20260619 02:49:59.864406 3718 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:59.858263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:43659 (local address 127.26.98.62:36183)
0619 02:49:59.858401 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.858403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.858415 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:49:59.858570 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.858572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.858594 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.858708 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.859119 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.859788 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.860646 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.860872 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.860940 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.864312 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:43659: BlockingRecv error: recv got EOF from 127.26.98.1:43659 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:00.005882 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.000020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:48879)
0619 02:50:00.000202 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.000217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.000310 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.000479 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.000480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.000680 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.000749 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.000751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.001564 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.001565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.002149 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.002151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.002673 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.002675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.002774 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.003517 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.003522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.005798 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
W20260619 02:50:00.006299 3718 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.000151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:48879 (local address 127.26.98.62:36183)
0619 02:50:00.000215 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.000218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.000289 (+ 71us) server_negotiation.cc:407] Connection header received
0619 02:50:00.000389 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.000389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.000403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.000451 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.000851 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.001411 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.002282 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.002495 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.002566 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.006230 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:48879: BlockingRecv error: recv got EOF from 127.26.98.2:48879 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:00.179894 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.174315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:56271)
0619 02:50:00.174474 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.174487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.174581 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.174734 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.174735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.174975 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.175042 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.175045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.175887 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.175889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.176510 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.176514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.176887 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.176889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.176962 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.177574 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.177579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.179801 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
W20260619 02:50:00.180061 3718 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.174451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:56271 (local address 127.26.98.62:36183)
0619 02:50:00.174560 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.174562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.174576 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:50:00.174638 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.174639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.174656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.174720 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.175149 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.175743 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.176608 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.176816 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.176883 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.179961 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:56271: BlockingRecv error: recv got EOF from 127.26.98.3:56271 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:00 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:00.870913 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.864777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:45855)
0619 02:50:00.864959 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.864970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.865025 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.865442 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.865443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.865641 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.865725 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.865728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.866552 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.866554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.867214 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.867218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.867731 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.867734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.867828 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.868548 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.868554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.870800 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
W20260619 02:50:00.871088 3722 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.864953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:45855 (local address 127.26.98.62:36183)
0619 02:50:00.865212 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.865215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.865254 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:50:00.865285 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.865286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.865310 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.865412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.865829 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.866419 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.867336 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.867574 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.867644 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.871003 (+ 3359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:45855: BlockingRecv error: recv got EOF from 127.26.98.1:45855 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":186,"threads_started":1}
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:01.013119 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.006756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:34539)
0619 02:50:01.007053 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.007066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.007160 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.007351 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.007351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.007550 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.007618 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.007621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.008694 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.008697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.009461 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.009467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.010013 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.010014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.010096 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.010690 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.010696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.013038 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":94,"threads_started":1}
W20260619 02:50:01.013346 3722 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.006968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:34539 (local address 127.26.98.62:36183)
0619 02:50:01.007033 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.007035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.007155 (+ 120us) server_negotiation.cc:407] Connection header received
0619 02:50:01.007249 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.007250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.007265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.007316 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.007696 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.008463 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.009634 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.009928 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.010011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.013273 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:34539: BlockingRecv error: recv got EOF from 127.26.98.2:34539 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:01.185734 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.180531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:37447)
0619 02:50:01.180710 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.180724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.180827 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.180978 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.180979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.181208 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.181269 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.181271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.181971 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.181973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.182529 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.182531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.182861 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.182862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.182969 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.183503 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.183508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.185658 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
W20260619 02:50:01.185947 3722 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.180695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:37447 (local address 127.26.98.62:36183)
0619 02:50:01.180762 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.180764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.180802 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:50:01.180896 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.180897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.180912 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.180964 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.181350 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.181888 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.182610 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.182796 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.182856 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.185817 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:37447: BlockingRecv error: recv got EOF from 127.26.98.3:37447 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:01 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:01.889657 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.871766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:49951)
0619 02:50:01.872103 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.872129 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.872254 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.872600 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.872602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.872840 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.872898 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.872900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.873646 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.873647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.874192 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.874194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.874571 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.874572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.874632 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.886999 (+ 12367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.887007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.889555 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":107,"threads_started":1}
W20260619 02:50:01.889796 3726 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.871921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:49951 (local address 127.26.98.62:36183)
0619 02:50:01.872063 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.872065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.872257 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:50:01.872331 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.872333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.872348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.872591 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.872989 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.873554 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.874291 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.874502 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.874565 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.889718 (+ 15153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:49951: BlockingRecv error: recv got EOF from 127.26.98.1:49951 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:02.020068 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.013958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:59697)
0619 02:50:02.014111 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.014123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.014205 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.014399 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.014400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.014612 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.014678 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.014680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.015497 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.015498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.016130 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.016133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.016575 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.016577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.016635 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.017311 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.017316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.019988 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
W20260619 02:50:02.020300 3726 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:02.013957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:59697 (local address 127.26.98.62:36183)
0619 02:50:02.014071 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.014073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.014247 (+ 174us) server_negotiation.cc:407] Connection header received
0619 02:50:02.014281 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.014282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.014307 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.014353 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.014784 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.015378 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.016248 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.016469 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.016543 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.020215 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:59697: BlockingRecv error: recv got EOF from 127.26.98.2:59697 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:02.193437 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.186469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:42647)
0619 02:50:02.186752 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.186764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.186849 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.187108 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.187110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.187315 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.187378 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.187380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.188210 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.188211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.188801 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.188803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.189178 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.189179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.189241 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.189933 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.189939 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.193340 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":173,"threads_started":1}
W20260619 02:50:02.193706 3726 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:02.186655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:42647 (local address 127.26.98.62:36183)
0619 02:50:02.186763 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.186765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.186828 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:50:02.186923 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.186924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.186939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.186996 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.187493 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.188096 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.188904 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.189103 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.189165 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.193632 (+ 4467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:42647: BlockingRecv error: recv got EOF from 127.26.98.3:42647 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:02 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:02.896719 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.890394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:44263)
0619 02:50:02.890630 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.890643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.890727 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.891000 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.891002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.891226 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.891290 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.891293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.892147 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.892150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.893251 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.893256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.893794 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.893795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.893851 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.894406 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.894412 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.896632 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
W20260619 02:50:02.896912 3730 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:02.890469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:44263 (local address 127.26.98.62:36183)
0619 02:50:02.890594 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.890597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.890822 (+ 225us) server_negotiation.cc:407] Connection header received
0619 02:50:02.890857 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.890857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.890874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.890991 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.891385 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.891992 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.893405 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.893657 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.893902 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.896823 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:44263: BlockingRecv error: recv got EOF from 127.26.98.1:44263 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:03.026698 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.020824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:57801)
0619 02:50:03.021036 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.021050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:03.021189 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:03.021374 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:03.021374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:03.021610 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:03.021675 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.021677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.022412 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.022413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.023015 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.023018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.023382 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.023383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.023437 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.024139 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:03.024144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:03.026613 (+ 2469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":124,"threads_started":1}
W20260619 02:50:03.026871 3730 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:03.020956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:57801 (local address 127.26.98.62:36183)
0619 02:50:03.021041 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:03.021044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:03.021180 (+ 136us) server_negotiation.cc:407] Connection header received
0619 02:50:03.021280 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:03.021281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:03.021297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:03.021362 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:03.021766 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.022323 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.023108 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.023314 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.023378 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.026790 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:57801: BlockingRecv error: recv got EOF from 127.26.98.2:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:03.203122 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.194439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:35369)
0619 02:50:03.194617 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.194628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:03.194831 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:03.195517 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:03.195518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:03.196512 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:03.196595 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.196601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.198128 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.198130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.198721 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.198724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.199885 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.199887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.199965 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.200797 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:03.200803 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:03.203043 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260619 02:50:03.203505 3730 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:03.194828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:35369 (local address 127.26.98.62:36183)
0619 02:50:03.194933 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:03.194936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:03.194950 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:50:03.195358 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:03.195359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:03.195378 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:03.195456 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:03.196949 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.197618 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.199262 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.199611 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.199778 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.203352 (+ 3574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:35369: BlockingRecv error: recv got EOF from 127.26.98.3:35369 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:03 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:03.904644 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.897455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:60037)
0619 02:50:03.897671 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.897688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:03.897901 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:03.898233 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:03.898234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:03.898674 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:03.898750 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.898755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.899678 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.899680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.900287 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.900289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.901078 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.901080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.901168 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.901905 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:03.901910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:03.904559 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
W20260619 02:50:03.904855 3734 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:03.897605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:60037 (local address 127.26.98.62:36183)
0619 02:50:03.897884 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:03.897886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:03.897899 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:50:03.898053 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:03.898054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:03.898069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:03.898169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:03.898980 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.899512 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.900449 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.900905 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.900992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.904762 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:60037: BlockingRecv error: recv got EOF from 127.26.98.1:60037 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":207,"threads_started":1}
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:04.035068 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.027479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:44417)
0619 02:50:04.027703 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.027721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.027829 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.028046 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.028048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.028277 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.028339 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.028340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.029191 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.029192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.029980 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.029985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.030506 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.030506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.030565 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.031416 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.031424 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.034939 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
W20260619 02:50:04.035125 3734 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:04.027671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:44417 (local address 127.26.98.62:36183)
0619 02:50:04.027798 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.027800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.027824 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:50:04.027894 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.027895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.027908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.027979 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.028482 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.029057 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.030107 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.030371 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.030441 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.035059 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:44417: BlockingRecv error: recv got EOF from 127.26.98.2:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:04.209738 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.203834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:57077)
0619 02:50:04.204051 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.204107 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.204184 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.204431 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.204432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.204652 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.204716 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.204718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.205569 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.205570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.206159 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.206162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.206699 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.206700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.206765 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.207370 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.207376 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.209622 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":114,"threads_started":1}
W20260619 02:50:04.209827 3734 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:04.204017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:57077 (local address 127.26.98.62:36183)
0619 02:50:04.204175 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.204178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.204257 (+ 79us) server_negotiation.cc:407] Connection header received
0619 02:50:04.204287 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.204288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.204304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.204365 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.204885 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.205440 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.206308 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.206567 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.206638 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.209761 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:57077: BlockingRecv error: recv got EOF from 127.26.98.3:57077 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:04 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:04.912035 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.905515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:54665)
0619 02:50:04.905724 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.905737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.905810 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.906043 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.906045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.906315 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.906378 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.906381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.907315 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.907316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.907957 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.907960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.908631 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.908633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.908714 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.909360 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.909365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.911909 (+ 2544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
W20260619 02:50:04.912180 3738 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:04.905636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:54665 (local address 127.26.98.62:36183)
0619 02:50:04.905784 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.905787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.905799 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:50:04.905909 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.905910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.905927 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.906024 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.906485 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.907144 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.908119 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.908367 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.908665 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.912070 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:54665: BlockingRecv error: recv got EOF from 127.26.98.1:54665 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:05.041458 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.035686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:43955)
0619 02:50:05.035913 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.035925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:05.035989 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:05.036127 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:05.036128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:05.036362 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:05.036413 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.036416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.037359 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.037361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.038073 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.038078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.038530 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.038531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.038588 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.039217 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:05.039222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:05.041340 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":132,"threads_started":1}
W20260619 02:50:05.041543 3738 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.035897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:43955 (local address 127.26.98.62:36183)
0619 02:50:05.035983 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.035985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.035995 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:05.036051 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.036051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.036064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.036112 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.036493 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.037190 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.038238 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.038455 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.038523 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.041467 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:43955: BlockingRecv error: recv got EOF from 127.26.98.2:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:05.217222 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.210508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:59871)
0619 02:50:05.210697 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.210719 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:05.210812 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:05.210975 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:05.210977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:05.211203 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:05.211276 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.211278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.212150 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.212152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.212893 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.212898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.213594 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.213597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.213679 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.214374 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:05.214380 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:05.217120 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
W20260619 02:50:05.217473 3738 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.210637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59871 (local address 127.26.98.62:36183)
0619 02:50:05.210736 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.210738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.210799 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:50:05.210875 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.210895 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.210909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.210959 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.211352 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.211993 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.213021 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.213378 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.213554 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.217385 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:59871: BlockingRecv error: recv got EOF from 127.26.98.3:59871 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:05 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:05.927064 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.913069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:55377)
0619 02:50:05.913294 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.913310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:05.913389 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:05.913592 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:05.913593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:05.913818 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:05.914048 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.914057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.915422 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.915424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.916000 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:05.916002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.916506 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.916508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:05.916600 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.924323 (+ 7723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:05.924332 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:05.926928 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":145,"threads_started":1}
W20260619 02:50:05.927126 3742 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.913202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:55377 (local address 127.26.98.62:36183)
0619 02:50:05.913413 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.913416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.913429 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:50:05.913463 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.913464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.913476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.913548 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.914266 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.915192 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.916112 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.916374 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.916443 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.927039 (+ 10596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:55377: BlockingRecv error: recv got EOF from 127.26.98.1:55377 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:06.048301 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.042204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:60413)
0619 02:50:06.042383 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.042396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.042471 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.042668 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.042669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.042968 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:06.043048 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.043051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.043968 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.043969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.044576 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.044578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.044962 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.044963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.045033 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.045619 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:06.045625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:06.048192 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260619 02:50:06.048511 3742 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:06.042417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:60413 (local address 127.26.98.62:36183)
0619 02:50:06.042489 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.042491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.042502 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:06.042542 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.042543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.042556 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.042649 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:06.043130 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.043855 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.044667 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.044885 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.044955 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.048402 (+ 3447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:60413: BlockingRecv error: recv got EOF from 127.26.98.2:60413 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:06.225001 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.218237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:33507)
0619 02:50:06.218405 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.218418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.218559 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.218868 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.218870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.219204 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:06.219272 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.219275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.220196 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.220205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.221170 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.221176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.221714 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.221715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.221776 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.222409 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:06.222414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:06.224885 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
W20260619 02:50:06.225068 3742 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:06.218314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:33507 (local address 127.26.98.62:36183)
0619 02:50:06.218435 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.218437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.218578 (+ 141us) server_negotiation.cc:407] Connection header received
0619 02:50:06.218618 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.218619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.218636 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.218710 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:06.219447 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.220033 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.221319 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.221541 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.221616 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.224995 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:33507: BlockingRecv error: recv got EOF from 127.26.98.3:33507 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:06 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:06.939239 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.927843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:36049)
0619 02:50:06.928037 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.928053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.928140 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.928377 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.928379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.928609 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:06.928668 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.928670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.929503 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.929504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.930132 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.930134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.930532 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.930533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.930605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.936758 (+ 6153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:06.936765 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:06.939135 (+ 2370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260619 02:50:06.939399 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:06.928002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:36049 (local address 127.26.98.62:36183)
0619 02:50:06.928198 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.928200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.928210 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:06.928237 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.928237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.928251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.928323 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:06.928783 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.929382 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.930230 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.930453 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.930524 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.939312 (+ 8788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:36049: BlockingRecv error: recv got EOF from 127.26.98.1:36049 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:07.054692 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.049038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:39367)
0619 02:50:07.049254 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.049270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.049389 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.049562 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.049563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.049816 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.049892 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.049894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.050815 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.050817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.051435 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.051438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.051795 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.051796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.051847 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.052385 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.052390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.054576 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
W20260619 02:50:07.054769 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:07.049115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:39367 (local address 127.26.98.62:36183)
0619 02:50:07.049260 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.049263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.049354 (+ 91us) server_negotiation.cc:407] Connection header received
0619 02:50:07.049469 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.049470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.049485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.049549 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.049952 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.050709 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.051522 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.051734 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.051794 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.054700 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:39367: BlockingRecv error: recv got EOF from 127.26.98.2:39367 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.3@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:07.231015 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.225655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.3:50487)
0619 02:50:07.225835 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.225848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.225927 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.226093 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.226094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.226296 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.226361 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.226364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.227137 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.227138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.227717 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.227720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.228131 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.228132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.228189 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.228787 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.228792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.230868 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
W20260619 02:50:07.231105 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:07.225800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:50487 (local address 127.26.98.62:36183)
0619 02:50:07.225853 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.225855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.225908 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:50:07.225973 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.225974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.225988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.226043 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.226440 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.227039 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.227821 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.228031 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.228097 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.231041 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.3:50487: BlockingRecv error: recv got EOF from 127.26.98.3:50487 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:07 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.1@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:07.946683 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.939942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.1:41177)
0619 02:50:07.940095 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.940106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.940168 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.940493 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.940496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.940727 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.940802 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.940804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.941869 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.941872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.942573 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.942577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.943199 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.943201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.943286 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.943974 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.943980 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.946600 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
W20260619 02:50:07.946839 3750 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:07.940055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:41177 (local address 127.26.98.62:36183)
0619 02:50:07.940168 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.940169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.940179 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:07.940208 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.940209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.940219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.940280 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.940993 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.941737 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.942668 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.943015 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.943115 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.946753 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.1:41177: BlockingRecv error: recv got EOF from 127.26.98.1:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Jun 19 02:50:08 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:08 dist-test-slave-0f9d krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.98.2@KRBTEST.COM for oryx/127.26.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:08.061205 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.055635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36183 (local address 127.26.98.2:32853)
0619 02:50:08.055869 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.055937 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.056012 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.056238 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.056240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.056448 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:08.056518 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.056520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.057346 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.057347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.057972 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.057974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.058405 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.058405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.058453 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.059185 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:08.059191 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:08.061126 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.98.62:36183: Server oryx/127.26.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
W20260619 02:50:08.061456 3750 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:08.055840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:32853 (local address 127.26.98.62:36183)
0619 02:50:08.055940 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.055942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.056005 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:50:08.056064 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.056065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.056076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.056139 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.056668 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.057221 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.058107 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.058321 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.058397 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.061375 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.98.2:32853: BlockingRecv error: recv got EOF from 127.26.98.2:32853 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:50:08.119881 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3275
I20260619 02:50:08.125674 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3427
I20260619 02:50:08.131333 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3560
I20260619 02:50:08.137310 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3205
2026-06-19T02:50:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17784 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:08Z Disabled control of system clock
I20260619 02:50:08.172909 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43439
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:50:08.286597 3755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:08.286863 3755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:08.286916 3755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:08.289166 3755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:08.289297 3755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:08.289312 3755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:08.289328 3755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:08.289342 3755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:08.291684 3755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:43439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:08.292146 3755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:08.292589 3755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:08.301064 3761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.301086 3760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.301103 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:08.301079 3755 server_base.cc:1034] running on GCE node
I20260619 02:50:08.301913 3755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:08.302268 3755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:08.303465 3755 hybrid_clock.cc:648] HybridClock initialized: now 1781837408303421 us; error 67 us; skew 500 ppm
I20260619 02:50:08.305467 3755 webserver.cc:460] Webserver started at http://127.26.98.62:37057/ using document root <none> and password file <none>
I20260619 02:50:08.305827 3755 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:08.305891 3755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:08.306017 3755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:08.307301 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "1017bcc3e0a247919f8d8c46e9c031a0"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "27ded012b1c2187f70240cb001c4520a"
server_key_iv: "dd516aef8ab54087639e90c63aab0d4b"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.307663 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "1017bcc3e0a247919f8d8c46e9c031a0"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "27ded012b1c2187f70240cb001c4520a"
server_key_iv: "dd516aef8ab54087639e90c63aab0d4b"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.309466 3755 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:50:08.310588 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.310945 3755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.311106 3755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "1017bcc3e0a247919f8d8c46e9c031a0"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "27ded012b1c2187f70240cb001c4520a"
server_key_iv: "dd516aef8ab54087639e90c63aab0d4b"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.311208 3755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:08.328318 3755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:08.328809 3755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:08.329123 3755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:08.335602 3755 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:43439
I20260619 02:50:08.335731 3821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:43439 every 8 connection(s)
I20260619 02:50:08.336175 3755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:08.337158 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.341523 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0: Bootstrap starting.
I20260619 02:50:08.342517 3822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.342875 3822 log.cc:824] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:08.343289 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3755
I20260619 02:50:08.343420 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:50:08.343643 27016 external_mini_cluster.cc:1428] Setting key 0df4fa389be832555a0e269a2bee7820
I20260619 02:50:08.344125 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0: No bootstrap required, opened a new log
I20260619 02:50:08.347749 3822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } }
I20260619 02:50:08.348044 3822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.348099 3822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1017bcc3e0a247919f8d8c46e9c031a0, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.348280 3822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } }
I20260619 02:50:08.348409 3822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:08.348449 3822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:08.348512 3822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.349517 3822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } }
I20260619 02:50:08.349649 3822 leader_election.cc:302] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1017bcc3e0a247919f8d8c46e9c031a0; no voters:
I20260619 02:50:08.349861 3822 leader_election.cc:288] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:08.349908 3827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:08.350040 3827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1017bcc3e0a247919f8d8c46e9c031a0, State: Running, Role: LEADER
I20260619 02:50:08.350163 3827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } }
I20260619 02:50:08.350207 3822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:08.350561 3827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } } }
I20260619 02:50:08.350775 3827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:08.351030 3827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1017bcc3e0a247919f8d8c46e9c031a0. Latest consensus state: current_term: 1 leader_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1017bcc3e0a247919f8d8c46e9c031a0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43439 } } }
I20260619 02:50:08.351163 3827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:08.351624 3836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:08.352290 3836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:08.354249 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.344494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.26.98.62:43439)
0619 02:50:08.344833 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.344899 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.344955 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:50:08.346316 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.346343 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.346621 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.346870 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.347183 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.349252 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.350977 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.351711 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.351847 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.353032 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.353037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.353038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:08.353469 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.353518 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:08.353533 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.353611 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.353829 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.353936 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":136,"threads_started":1}
I20260619 02:50:08.354596 3836 catalog_manager.cc:1347] Generated new cluster ID: c3fc87d4599c44cb959af52dc5b7ab9e
I20260619 02:50:08.354650 3836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:08.361812 3836 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:08.362530 3836 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:08.365368 3836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1017bcc3e0a247919f8d8c46e9c031a0: Generated new TSK 0
I20260619 02:50:08.365629 3836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:08.372288 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--builtin_ntp_servers=127.26.98.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:50:08.485707 3846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:08.486022 3846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:08.486052 3846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:08.487833 3846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:08.487911 3846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:08.487944 3846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:50:08.491000 3846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:08.491389 3846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:08.491890 3846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:08.499742 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.499753 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.499925 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:08.500483 3846 server_base.cc:1034] running on GCE node
I20260619 02:50:08.500713 3846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:08.501205 3846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:08.502509 3846 hybrid_clock.cc:648] HybridClock initialized: now 1781837408502457 us; error 48 us; skew 500 ppm
I20260619 02:50:08.504426 3846 webserver.cc:460] Webserver started at http://127.26.98.1:35703/ using document root <none> and password file <none>
I20260619 02:50:08.504748 3846 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:08.504810 3846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:08.504988 3846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:08.506420 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "226a868914704f7d9e76490f93126505"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "b0983856ba5306a14c9fb2d6fbc544f7"
server_key_iv: "6ed399ecebc5408d23cfb40b12d0994f"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.506842 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "226a868914704f7d9e76490f93126505"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "b0983856ba5306a14c9fb2d6fbc544f7"
server_key_iv: "6ed399ecebc5408d23cfb40b12d0994f"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.508533 3846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.509466 3860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.509693 3846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.509830 3846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "226a868914704f7d9e76490f93126505"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "b0983856ba5306a14c9fb2d6fbc544f7"
server_key_iv: "6ed399ecebc5408d23cfb40b12d0994f"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.509979 3846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:08.531585 3846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:08.532024 3846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:08.532171 3846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:08.532430 3846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:08.532825 3846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:08.532862 3846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.532888 3846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:08.532902 3846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.540429 3846 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:37807
I20260619 02:50:08.540495 3973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:37807 every 8 connection(s)
I20260619 02:50:08.540925 3846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:08.548235 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.541949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35831 (local address 127.26.98.62:43439)
0619 02:50:08.542043 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.542045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.542335 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:50:08.543073 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.543075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.543098 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.543160 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.544398 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.544988 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.546188 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.546393 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.546458 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.547896 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.547898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.547898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:08.547906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.547924 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:08.547929 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.547980 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.548067 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.548127 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:50:08.548573 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.541916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43439 (local address 127.26.98.1:35831)
0619 02:50:08.542187 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.542231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.542948 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.543265 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.543270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.543285 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:08.544253 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.544263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.545103 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.545104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.546072 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.546076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.546708 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.546712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.546974 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.547693 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:08.547693 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:08.547735 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:08.547773 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:08.547987 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:08.547993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:08.547997 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.548067 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.548220 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
I20260619 02:50:08.549428 3974 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43439
I20260619 02:50:08.549563 3974 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:08.549818 3974 heartbeater.cc:511] Master 127.26.98.62:43439 requested a full tablet report, sending...
I20260619 02:50:08.550505 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3846
I20260619 02:50:08.550597 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:08.550565 3786 ts_manager.cc:194] Registered new tserver with Master: 226a868914704f7d9e76490f93126505 (127.26.98.1:37807)
I20260619 02:50:08.550849 27016 external_mini_cluster.cc:1428] Setting key 9ab2127c90792c8b66b598fcd1ef6edd
I20260619 02:50:08.551534 3786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.1:35831
I20260619 02:50:08.552351 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--builtin_ntp_servers=127.26.98.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:50:08.651021 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:08.651352 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:08.651386 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:08.653096 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:08.653185 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:08.653223 3977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:50:08.655184 3977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:08.655545 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:08.655952 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:08.661974 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.662024 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.662190 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:08.663049 3977 server_base.cc:1034] running on GCE node
I20260619 02:50:08.663529 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:08.663997 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:08.665231 3977 hybrid_clock.cc:648] HybridClock initialized: now 1781837408665153 us; error 95 us; skew 500 ppm
I20260619 02:50:08.666996 3977 webserver.cc:460] Webserver started at http://127.26.98.2:39089/ using document root <none> and password file <none>
I20260619 02:50:08.667335 3977 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:08.667405 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:08.667557 3977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:08.668872 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "df6a73bbdc644d3a87ef9b392ae102c6"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "a6f6098b4847b6f209c70678612085b4"
server_key_iv: "72f6fbd36b5b230062f40af3c7476c24"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.669277 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "df6a73bbdc644d3a87ef9b392ae102c6"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "a6f6098b4847b6f209c70678612085b4"
server_key_iv: "72f6fbd36b5b230062f40af3c7476c24"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.670980 3977 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:50:08.672060 3991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.672323 3977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.672436 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "df6a73bbdc644d3a87ef9b392ae102c6"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "a6f6098b4847b6f209c70678612085b4"
server_key_iv: "72f6fbd36b5b230062f40af3c7476c24"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.672534 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:08.683096 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:08.683527 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:08.683692 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:08.683979 3977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:08.684377 3977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:08.684413 3977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.684446 3977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:08.684464 3977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.691556 3977 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:35785
I20260619 02:50:08.691622 4104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:35785 every 8 connection(s)
I20260619 02:50:08.692006 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:08.697906 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.692874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:45927 (local address 127.26.98.62:43439)
0619 02:50:08.692936 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.692938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.693211 (+ 273us) server_negotiation.cc:407] Connection header received
0619 02:50:08.693859 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.693861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.693875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.693939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.694740 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695334 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.696418 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.696620 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.696680 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697609 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.697610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.697610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:08.697617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.697636 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:08.697640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.697683 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.697772 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.697805 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:50:08.698161 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.692863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43439 (local address 127.26.98.2:45927)
0619 02:50:08.693071 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.693113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.693740 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.694029 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.694033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.694043 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:08.694637 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.694641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695429 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.695430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.696324 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.696327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.696696 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.696697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.696844 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697453 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:08.697454 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:08.697488 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:08.697518 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:08.697691 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:08.697698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:08.697703 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.697772 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.697900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260619 02:50:08.698738 4105 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43439
I20260619 02:50:08.698851 4105 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:08.699079 4105 heartbeater.cc:511] Master 127.26.98.62:43439 requested a full tablet report, sending...
I20260619 02:50:08.699474 3786 ts_manager.cc:194] Registered new tserver with Master: df6a73bbdc644d3a87ef9b392ae102c6 (127.26.98.2:35785)
I20260619 02:50:08.699865 3786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.2:45927
I20260619 02:50:08.700594 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 3977
I20260619 02:50:08.700700 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:08.700911 27016 external_mini_cluster.cc:1428] Setting key 8cdc23a1626d9cd823ed2c524b0aaf9e
I20260619 02:50:08.701853 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--builtin_ntp_servers=127.26.98.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:50:08.803058 4108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:08.803450 4108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:08.803490 4108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:08.805035 4108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:08.805097 4108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:08.805131 4108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:50:08.807113 4108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:08.807435 4108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:08.807811 4108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:08.813982 4113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.813966 4114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:08.813975 4116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:08.814455 4108 server_base.cc:1034] running on GCE node
I20260619 02:50:08.814735 4108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:08.815272 4108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:08.816431 4108 hybrid_clock.cc:648] HybridClock initialized: now 1781837408816381 us; error 76 us; skew 500 ppm
I20260619 02:50:08.818480 4108 webserver.cc:460] Webserver started at http://127.26.98.3:36413/ using document root <none> and password file <none>
I20260619 02:50:08.818775 4108 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:08.818835 4108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:08.819053 4108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:08.820469 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "9ffb909febff4d6e80aa4faed7a67896"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "4f2c1e452bda2ceb4795c70bc9eec351"
server_key_iv: "2758dea38361ee3e12b557a765ea8c8c"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.820919 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffb909febff4d6e80aa4faed7a67896"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "4f2c1e452bda2ceb4795c70bc9eec351"
server_key_iv: "2758dea38361ee3e12b557a765ea8c8c"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.822592 4108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260619 02:50:08.823725 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.824136 4108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.824257 4108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "9ffb909febff4d6e80aa4faed7a67896"
format_stamp: "Formatted at 2026-06-19 02:50:08 on dist-test-slave-0f9d"
server_key: "4f2c1e452bda2ceb4795c70bc9eec351"
server_key_iv: "2758dea38361ee3e12b557a765ea8c8c"
server_key_version: "encryptionkey@0"
I20260619 02:50:08.824357 4108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:08.844691 4108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:08.845129 4108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:08.845279 4108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:08.845554 4108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:08.845958 4108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:08.845995 4108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.846030 4108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:08.846056 4108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.853976 4108 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:34761
I20260619 02:50:08.854063 4235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:34761 every 8 connection(s)
I20260619 02:50:08.854482 4108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:50:08.859148 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4108
I20260619 02:50:08.859238 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:08.859440 27016 external_mini_cluster.cc:1428] Setting key 6506346f01f006c16dbfed21e3c4e97b
I20260619 02:50:08.862824 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.855754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:33965 (local address 127.26.98.62:43439)
0619 02:50:08.855882 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.855885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.856220 (+ 335us) server_negotiation.cc:407] Connection header received
0619 02:50:08.857330 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.857332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.857348 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.857453 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.858497 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.859179 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.861028 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.861240 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.861309 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.862536 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.862538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.862538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:08.862550 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.862569 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:08.862574 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.862634 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.862706 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.862743 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:50:08.863173 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.855695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43439 (local address 127.26.98.3:33965)
0619 02:50:08.855990 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.856062 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.857050 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.857631 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.857638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.857652 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:08.858320 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.858327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.859391 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.859393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.860837 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.860841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.861384 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.861386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.861600 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.862332 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:08.862333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:08.862373 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:08.862405 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:08.862641 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:08.862645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:08.862648 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.862703 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.862829 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":118,"threads_started":1}
I20260619 02:50:08.863708 4236 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43439
I20260619 02:50:08.863833 4236 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:08.864063 4236 heartbeater.cc:511] Master 127.26.98.62:43439 requested a full tablet report, sending...
I20260619 02:50:08.864607 3786 ts_manager.cc:194] Registered new tserver with Master: 9ffb909febff4d6e80aa4faed7a67896 (127.26.98.3:34761)
I20260619 02:50:08.865077 3786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.3:33965
I20260619 02:50:08.872539 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:50:08.875262 3826 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:08.874650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.26.98.62:43439)
0619 02:50:08.874741 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.874744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.874758 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:50:08.874827 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.874828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.874838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.874936 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.875171 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51786: BlockingRecv error: recv got EOF from 127.0.0.1:51786 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:50:08.875480 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3846
I20260619 02:50:08.880613 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3977
I20260619 02:50:08.885578 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4108
I20260619 02:50:08.890460 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 3755
2026-06-19T02:50:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (748 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:08Z Disabled control of system clock
I20260619 02:50:08.920948 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34275
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:34275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:09.025789 4247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:09.026135 4247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:09.026165 4247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:09.027849 4247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:09.027922 4247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:09.027935 4247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:09.027947 4247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:09.027959 4247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:09.029771 4247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:34275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:34275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:09.030112 4247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:09.030440 4247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:09.035965 4252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:09.036144 4247 server_base.cc:1034] running on GCE node
W20260619 02:50:09.035962 4255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.035962 4253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:09.036533 4247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:09.036895 4247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:09.038095 4247 hybrid_clock.cc:648] HybridClock initialized: now 1781837409038075 us; error 45 us; skew 500 ppm
I20260619 02:50:09.039777 4247 webserver.cc:460] Webserver started at http://127.26.98.62:40501/ using document root <none> and password file <none>
I20260619 02:50:09.040076 4247 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:09.040138 4247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:09.040254 4247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:09.041577 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "476740a96b774818b6d86ecb32e01a4c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "76513c15dca26a5f3155d3e60aabb792"
server_key_iv: "6bfa8e178cf55ed6c9c4082015912fd2"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.041986 4247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "476740a96b774818b6d86ecb32e01a4c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "76513c15dca26a5f3155d3e60aabb792"
server_key_iv: "6bfa8e178cf55ed6c9c4082015912fd2"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.043797 4247 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260619 02:50:09.045020 4261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.045395 4247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:09.045516 4247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "476740a96b774818b6d86ecb32e01a4c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "76513c15dca26a5f3155d3e60aabb792"
server_key_iv: "6bfa8e178cf55ed6c9c4082015912fd2"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.045620 4247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:09.050199 4247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:09.050613 4247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:09.050769 4247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:09.056012 4247 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:34275
I20260619 02:50:09.056064 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:34275 every 8 connection(s)
I20260619 02:50:09.056489 4247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:09.057451 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4247
I20260619 02:50:09.057567 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:50:09.057581 4314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:09.057816 27016 external_mini_cluster.cc:1428] Setting key 5c7b163ff68840751b7ff9cc20819db8
I20260619 02:50:09.061583 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.058685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.26.98.62:34275)
0619 02:50:09.059016 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.059029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.059161 (+ 132us) server_negotiation.cc:407] Connection header received
0619 02:50:09.059958 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.059972 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.060055 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.060278 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:09.061048 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:09.061053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:09.061053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:09.061059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:09.061113 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:09.061129 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:09.061211 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.061278 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.061345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260619 02:50:09.062258 4314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: Bootstrap starting.
I20260619 02:50:09.063108 4314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:09.063511 4314 log.cc:824] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:09.064471 4314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: No bootstrap required, opened a new log
I20260619 02:50:09.067005 4314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } }
I20260619 02:50:09.067286 4314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:09.067314 4314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476740a96b774818b6d86ecb32e01a4c, State: Initialized, Role: FOLLOWER
I20260619 02:50:09.067469 4314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } }
I20260619 02:50:09.067559 4314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:09.067591 4314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:09.067651 4314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.068636 4314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } }
I20260619 02:50:09.068787 4314 leader_election.cc:302] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476740a96b774818b6d86ecb32e01a4c; no voters:
I20260619 02:50:09.069039 4314 leader_election.cc:288] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:09.069356 4314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:09.069442 4319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:09.069628 4319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [term 1 LEADER]: Becoming Leader. State: Replica: 476740a96b774818b6d86ecb32e01a4c, State: Running, Role: LEADER
I20260619 02:50:09.069779 4319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } }
I20260619 02:50:09.070338 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 476740a96b774818b6d86ecb32e01a4c. Latest consensus state: current_term: 1 leader_uuid: "476740a96b774818b6d86ecb32e01a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } } }
I20260619 02:50:09.070459 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:09.070539 4320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476740a96b774818b6d86ecb32e01a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476740a96b774818b6d86ecb32e01a4c" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 34275 } } }
I20260619 02:50:09.070632 4320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c [sys.catalog]: This master's current role is: LEADER
W20260619 02:50:09.071496 4334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:09.071571 4334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:09.071648 4330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:09.071856 4330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:09.073915 4330 catalog_manager.cc:1347] Generated new cluster ID: ae0047f40ba649d2a1513b8685b921f6
I20260619 02:50:09.074004 4330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:09.083809 4330 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:09.084158 4330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:09.090943 4330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 476740a96b774818b6d86ecb32e01a4c: Generated new TSK 0
I20260619 02:50:09.091236 4330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:09.096230 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--builtin_ntp_servers=127.26.98.20:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:09.217845 4338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:09.218147 4338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:09.218178 4338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:09.220213 4338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:09.220307 4338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:09.220350 4338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:50:09.222575 4338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:09.222940 4338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:09.223331 4338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:09.229319 4346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.229318 4343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.229449 4344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:09.229852 4338 server_base.cc:1034] running on GCE node
I20260619 02:50:09.230096 4338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:09.230441 4338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:09.231652 4338 hybrid_clock.cc:648] HybridClock initialized: now 1781837409231633 us; error 35 us; skew 500 ppm
I20260619 02:50:09.233685 4338 webserver.cc:460] Webserver started at http://127.26.98.1:37565/ using document root <none> and password file <none>
I20260619 02:50:09.234094 4338 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:09.234200 4338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:09.234370 4338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:09.236258 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "c638e473c46440eb839d3de35aa6674c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "25e1f782a9bc826e72cd26d58ab41864"
server_key_iv: "aba74c2803ead2df32ee19fdad81294e"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.236797 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "c638e473c46440eb839d3de35aa6674c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "25e1f782a9bc826e72cd26d58ab41864"
server_key_iv: "aba74c2803ead2df32ee19fdad81294e"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.238821 4338 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:50:09.240082 4352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.240366 4338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:09.240486 4338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "c638e473c46440eb839d3de35aa6674c"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "25e1f782a9bc826e72cd26d58ab41864"
server_key_iv: "aba74c2803ead2df32ee19fdad81294e"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.240588 4338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:09.245023 4338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:09.245391 4338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:09.245536 4338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:09.245795 4338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:09.246163 4338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:09.246199 4338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.246234 4338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:09.246260 4338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.255430 4338 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:33009
I20260619 02:50:09.255577 4465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:33009 every 8 connection(s)
I20260619 02:50:09.256300 4338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:09.260597 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.257502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:60451 (local address 127.26.98.62:34275)
0619 02:50:09.257635 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.257637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.257821 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:50:09.258377 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.258379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.258388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.258496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:09.260199 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:09.260202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:09.260202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:09.260208 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:09.260233 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:09.260239 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:09.260320 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.260502 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.260521 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 02:50:09.260887 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.257381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34275 (local address 127.26.98.1:60451)
0619 02:50:09.257663 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.257701 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:09.258238 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:09.258694 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:09.258700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:09.258714 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:09.259908 (+ 1194us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:09.259908 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:09.259953 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:09.260002 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:09.260369 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:09.260377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:09.260381 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:09.260475 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:50:09.260561 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
I20260619 02:50:09.261768 4466 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34275
I20260619 02:50:09.262007 4466 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:09.262341 4466 heartbeater.cc:511] Master 127.26.98.62:34275 requested a full tablet report, sending...
I20260619 02:50:09.263221 4278 ts_manager.cc:194] Registered new tserver with Master: c638e473c46440eb839d3de35aa6674c (127.26.98.1:33009)
I20260619 02:50:09.265712 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4338
I20260619 02:50:09.265837 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:09.266383 27016 external_mini_cluster.cc:1428] Setting key 0fcbdda88396a84458e70cffa09e324e
I20260619 02:50:09.268422 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--builtin_ntp_servers=127.26.98.20:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:09.371222 4469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:09.371551 4469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:09.371582 4469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:09.373252 4469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:09.373334 4469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:09.373370 4469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:50:09.375967 4469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:09.376411 4469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:09.376963 4469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:09.383059 4475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.383029 4474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.383179 4477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:09.383404 4469 server_base.cc:1034] running on GCE node
I20260619 02:50:09.383855 4469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:09.384253 4469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:09.385461 4469 hybrid_clock.cc:648] HybridClock initialized: now 1781837409385430 us; error 50 us; skew 500 ppm
I20260619 02:50:09.387331 4469 webserver.cc:460] Webserver started at http://127.26.98.2:42403/ using document root <none> and password file <none>
I20260619 02:50:09.387646 4469 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:09.387709 4469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:09.387861 4469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:09.389262 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "83bbfa4382fd4c839aa27606ed881823"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "94985631559989d224a7a06e362037b0"
server_key_iv: "eeafe0b4bc9e1953520bc5581ccafd2b"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.389700 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "83bbfa4382fd4c839aa27606ed881823"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "94985631559989d224a7a06e362037b0"
server_key_iv: "eeafe0b4bc9e1953520bc5581ccafd2b"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.391383 4469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 02:50:09.392552 4483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.392832 4469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:09.392953 4469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "83bbfa4382fd4c839aa27606ed881823"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "94985631559989d224a7a06e362037b0"
server_key_iv: "eeafe0b4bc9e1953520bc5581ccafd2b"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.393054 4469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:09.398073 4469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:09.398439 4469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:09.398583 4469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:09.398794 4469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:09.399207 4469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:09.399246 4469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.399279 4469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:09.399303 4469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.407884 4469 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:42291
I20260619 02:50:09.407924 4596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:42291 every 8 connection(s)
I20260619 02:50:09.408370 4469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:09.412123 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.409313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:54977 (local address 127.26.98.62:34275)
0619 02:50:09.409374 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.409376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.409632 (+ 256us) server_negotiation.cc:407] Connection header received
0619 02:50:09.410212 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.410214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.410221 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.410292 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:09.411763 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:09.411765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:09.411765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:09.411796 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:09.411817 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:09.411822 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:09.411895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.412023 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.412039 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:50:09.412386 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.409245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34275 (local address 127.26.98.2:54977)
0619 02:50:09.409465 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.409515 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:09.410085 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:09.410377 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:09.410382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:09.410394 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:09.411546 (+ 1152us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:09.411546 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:09.411589 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:09.411636 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:09.411928 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:09.411935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:09.411939 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:09.412049 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:50:09.412129 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260619 02:50:09.413058 4597 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34275
I20260619 02:50:09.413187 4597 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:09.413365 4597 heartbeater.cc:511] Master 127.26.98.62:34275 requested a full tablet report, sending...
I20260619 02:50:09.413854 4278 ts_manager.cc:194] Registered new tserver with Master: 83bbfa4382fd4c839aa27606ed881823 (127.26.98.2:42291)
I20260619 02:50:09.417737 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4469
I20260619 02:50:09.417842 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:09.418020 27016 external_mini_cluster.cc:1428] Setting key beb27c1b7fb3a3f80e8d8a441c0a1d9a
I20260619 02:50:09.419193 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--builtin_ntp_servers=127.26.98.20:46493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:09.523756 4600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:09.524067 4600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:09.524097 4600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:09.525774 4600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:09.525844 4600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:09.525879 4600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:50:09.527704 4600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:09.528003 4600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:09.528390 4600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:09.534572 4606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.534586 4608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:09.534672 4605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:09.535238 4600 server_base.cc:1034] running on GCE node
I20260619 02:50:09.535499 4600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:09.535900 4600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:09.537164 4600 hybrid_clock.cc:648] HybridClock initialized: now 1781837409537148 us; error 64 us; skew 500 ppm
I20260619 02:50:09.539006 4600 webserver.cc:460] Webserver started at http://127.26.98.3:37301/ using document root <none> and password file <none>
I20260619 02:50:09.539307 4600 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:09.539366 4600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:09.539505 4600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:09.540803 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "fda9016d015349ffbae0b3b55a5d3b33"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "6a6b436efd7bfb932fba7ea8263ed216"
server_key_iv: "fe971a06ddfc159ec70e735325971b61"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.541198 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "fda9016d015349ffbae0b3b55a5d3b33"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "6a6b436efd7bfb932fba7ea8263ed216"
server_key_iv: "fe971a06ddfc159ec70e735325971b61"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.542814 4600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:09.543915 4614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.544144 4600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:09.544256 4600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "fda9016d015349ffbae0b3b55a5d3b33"
format_stamp: "Formatted at 2026-06-19 02:50:09 on dist-test-slave-0f9d"
server_key: "6a6b436efd7bfb932fba7ea8263ed216"
server_key_iv: "fe971a06ddfc159ec70e735325971b61"
server_key_version: "encryptionkey@0"
I20260619 02:50:09.544363 4600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:09.548825 4600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:09.549206 4600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:09.549368 4600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:09.549626 4600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:09.550001 4600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:09.550035 4600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.550069 4600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:09.550102 4600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.558123 4600 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:45649
I20260619 02:50:09.558179 4727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:45649 every 8 connection(s)
I20260619 02:50:09.558610 4600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:50:09.562533 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.559643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:57229 (local address 127.26.98.62:34275)
0619 02:50:09.559721 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.559723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.559996 (+ 273us) server_negotiation.cc:407] Connection header received
0619 02:50:09.560564 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.560565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.560572 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.560640 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:09.562207 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:09.562209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:09.562209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:09.562214 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:09.562234 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:09.562239 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:09.562311 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.562433 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.562452 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:50:09.562780 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.559592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:34275 (local address 127.26.98.3:57229)
0619 02:50:09.559833 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.559883 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:09.560445 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:09.560709 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:09.560714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:09.560724 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:09.561987 (+ 1263us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:09.561988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:09.562038 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:09.562081 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:09.562340 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:09.562346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:09.562350 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:09.562438 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:50:09.562524 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260619 02:50:09.563405 4728 heartbeater.cc:347] Connected to a master server at 127.26.98.62:34275
I20260619 02:50:09.563535 4728 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:09.563728 4728 heartbeater.cc:511] Master 127.26.98.62:34275 requested a full tablet report, sending...
I20260619 02:50:09.564340 4278 ts_manager.cc:194] Registered new tserver with Master: fda9016d015349ffbae0b3b55a5d3b33 (127.26.98.3:45649)
I20260619 02:50:09.566026 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4600
I20260619 02:50:09.566115 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:09.566335 27016 external_mini_cluster.cc:1428] Setting key 40416944d751d1b9059054820c14f83c
I20260619 02:50:09.567502 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:50:09.569972 4317 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:09.569293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.26.98.62:34275)
0619 02:50:09.569348 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.569374 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.569568 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:50:09.569634 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.569634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.569642 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.569693 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:09.569881 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59398: BlockingRecv error: recv got EOF from 127.0.0.1:59398 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:50:09.570048 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4338
I20260619 02:50:09.574813 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4469
I20260619 02:50:09.579849 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4600
I20260619 02:50:09.585521 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4247
2026-06-19T02:50:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (695 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:09 dist-test-slave-0f9d krb5kdc[4738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:09 dist-test-slave-0f9d krb5kdc[4738](info): set up 2 sockets
Jun 19 02:50:09 dist-test-slave-0f9d krb5kdc[4738](info): commencing operation
krb5kdc: starting...
W20260619 02:50:10.601063 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:50:10 dist-test-slave-0f9d krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:50:10.747423 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45339
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:10.852945 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:10.853344 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:10.853386 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:10.855607 4751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:10.855720 4751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:10.855751 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:10.855772 4751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:10.855788 4751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:10.858604 4751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:10.859004 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:10.859264 4751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:10.866101 4756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:10.866074 4759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:10.866039 4757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:10.866124 4751 server_base.cc:1034] running on GCE node
I20260619 02:50:10.866596 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:10.867031 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:10.868403 4751 hybrid_clock.cc:648] HybridClock initialized: now 1781837410868370 us; error 45 us; skew 500 ppm
Jun 19 02:50:10 dist-test-slave-0f9d krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837410, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:10.872742 4751 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:50:10.873505 4751 webserver.cc:460] Webserver started at http://127.26.98.62:34007/ using document root <none> and password file <none>
I20260619 02:50:10.873958 4751 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:10.874029 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:10.874188 4751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:10.875876 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "87e833479beb4b7fbd1bb7bce1b43128"
format_stamp: "Formatted at 2026-06-19 02:50:10 on dist-test-slave-0f9d"
server_key: "8bfdab1633161276c5bade4158f0033a"
server_key_iv: "ac3fa7ac12f08287e458b47bc7ff74a2"
server_key_version: "encryptionkey@0"
I20260619 02:50:10.876447 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "87e833479beb4b7fbd1bb7bce1b43128"
format_stamp: "Formatted at 2026-06-19 02:50:10 on dist-test-slave-0f9d"
server_key: "8bfdab1633161276c5bade4158f0033a"
server_key_iv: "ac3fa7ac12f08287e458b47bc7ff74a2"
server_key_version: "encryptionkey@0"
I20260619 02:50:10.878225 4751 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:50:10.879843 4766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:10.880280 4751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:10.880463 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "87e833479beb4b7fbd1bb7bce1b43128"
format_stamp: "Formatted at 2026-06-19 02:50:10 on dist-test-slave-0f9d"
server_key: "8bfdab1633161276c5bade4158f0033a"
server_key_iv: "ac3fa7ac12f08287e458b47bc7ff74a2"
server_key_version: "encryptionkey@0"
I20260619 02:50:10.880579 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:10.926793 4751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:10.929383 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:10.929558 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:10.934791 4751 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45339
I20260619 02:50:10.934916 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45339 every 8 connection(s)
I20260619 02:50:10.935272 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:10.935606 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4751
I20260619 02:50:10.935729 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:50:10.935976 27016 external_mini_cluster.cc:1428] Setting key a1d7813c193c385cef90f46b72da2910
I20260619 02:50:10.936591 4819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:10.941905 4819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: Bootstrap starting.
I20260619 02:50:10.942623 4819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:10.942982 4819 log.cc:824] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:50:10 dist-test-slave-0f9d krb5kdc[4738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:50:10.943979 4819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: No bootstrap required, opened a new log
I20260619 02:50:10.946626 4819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } }
I20260619 02:50:10.946859 4819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:10.946943 4819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e833479beb4b7fbd1bb7bce1b43128, State: Initialized, Role: FOLLOWER
I20260619 02:50:10.947118 4819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } }
I20260619 02:50:10.947216 4819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:10.947261 4819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:10.947324 4819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:10.948148 4819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } }
I20260619 02:50:10.948274 4819 leader_election.cc:302] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87e833479beb4b7fbd1bb7bce1b43128; no voters:
I20260619 02:50:10.948499 4819 leader_election.cc:288] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:10.948583 4824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:10.948734 4824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [term 1 LEADER]: Becoming Leader. State: Replica: 87e833479beb4b7fbd1bb7bce1b43128, State: Running, Role: LEADER
I20260619 02:50:10.948865 4819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:10.948891 4824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } }
I20260619 02:50:10.949213 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:10.936688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.26.98.62:45339)
0619 02:50:10.936902 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:10.936913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:10.936948 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:50:10.937555 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:10.937567 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:10.937635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:10.937818 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:10.938523 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:10.939617 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:10.940437 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:10.940953 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:10.941069 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:10.944438 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:10.944443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:10.944444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:10.944458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:10.946748 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:10.947411 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:10.947416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:10.947417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:10.947536 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:10.947757 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:10.947758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:10.947759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:10.948075 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:10.948231 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:10.948643 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 02:50:10.948824 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260619 02:50:10.950516 4825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87e833479beb4b7fbd1bb7bce1b43128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } } }
I20260619 02:50:10.950525 4826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87e833479beb4b7fbd1bb7bce1b43128. Latest consensus state: current_term: 1 leader_uuid: "87e833479beb4b7fbd1bb7bce1b43128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e833479beb4b7fbd1bb7bce1b43128" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45339 } } }
I20260619 02:50:10.950630 4825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:10.950664 4826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128 [sys.catalog]: This master's current role is: LEADER
W20260619 02:50:10.951467 4838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:10.951536 4838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:10.951586 4839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:10.951753 4839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:10.953653 4839 catalog_manager.cc:1347] Generated new cluster ID: 0c7e089ada0c455bb0a916ef90aef7dc
I20260619 02:50:10.953722 4839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:10.975478 4839 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:10.976318 4839 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:10.981280 4839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: Generated new TSK 0
I20260619 02:50:10.981586 4839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:50:11.019954 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.148039 4845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.148344 4845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.148378 4845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.150003 4845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.150089 4845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.150128 4845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:50:11.152215 4845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:11.152863 4845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.153220 4845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.161005 4851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.161109 4845 server_base.cc:1034] running on GCE node
W20260619 02:50:11.161307 4850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.161412 4853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.161685 4845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.162107 4845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.163353 4845 hybrid_clock.cc:648] HybridClock initialized: now 1781837411163308 us; error 64 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.166666 4845 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:50:11.167276 4845 webserver.cc:460] Webserver started at http://127.26.98.1:41777/ using document root <none> and password file <none>
I20260619 02:50:11.167524 4845 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.167575 4845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.167712 4845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.169049 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "983d893ab4db42468308351d93e7f0b2"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "33ae0e42334cfed472f35fa4f8ee4d78"
server_key_iv: "61ee5b8471cc8ab0870cf3a94ec5b09a"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.169445 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "983d893ab4db42468308351d93e7f0b2"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "33ae0e42334cfed472f35fa4f8ee4d78"
server_key_iv: "61ee5b8471cc8ab0870cf3a94ec5b09a"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.171355 4845 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:50:11.172874 4860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.173211 4845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:11.173331 4845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "983d893ab4db42468308351d93e7f0b2"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "33ae0e42334cfed472f35fa4f8ee4d78"
server_key_iv: "61ee5b8471cc8ab0870cf3a94ec5b09a"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.173418 4845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.204165 4845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.207525 4845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.207705 4845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.208003 4845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:11.208374 4845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:11.208408 4845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.208442 4845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:11.208467 4845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.216114 4845 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:40101
I20260619 02:50:11.216220 4973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:40101 every 8 connection(s)
I20260619 02:50:11.216603 4845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:11.219158 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4845
I20260619 02:50:11.219302 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:11.219537 27016 external_mini_cluster.cc:1428] Setting key 198424681966d4fe58d9758ed2c46752
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:50:11.228739 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.217579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:46525 (local address 127.26.98.62:45339)
0619 02:50:11.217675 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.217677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.217861 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:50:11.218569 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.218572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.218588 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.218688 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.220206 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.220769 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.222156 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.222386 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.222448 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.225536 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.225538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.225539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.225556 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.227146 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.227646 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.227648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.227648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.227708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.227956 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.227957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.227957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.228095 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.228155 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.228488 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.228611 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:50:11.228935 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.217514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45339 (local address 127.26.98.1:46525)
0619 02:50:11.217691 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.217746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.218370 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.218858 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.218864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.219807 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:11.220082 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.220091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.220869 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.220870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.221998 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.222004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.222471 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.222473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.222657 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.223416 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:11.223429 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:11.225392 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:11.227249 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.227265 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.227270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.227561 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.227768 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.227768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.227769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.227896 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.228162 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:11.228169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:11.228347 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.228455 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.228669 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260619 02:50:11.229527 4974 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45339
I20260619 02:50:11.229673 4974 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:11.229910 4974 heartbeater.cc:511] Master 127.26.98.62:45339 requested a full tablet report, sending...
I20260619 02:50:11.230717 4783 ts_manager.cc:194] Registered new tserver with Master: 983d893ab4db42468308351d93e7f0b2 (127.26.98.1:40101)
I20260619 02:50:11.231897 4783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:46525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:50:11.252123 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.361513 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.361867 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.361909 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.363713 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.363802 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.363852 4979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:50:11.367324 4979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:11.367926 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.368247 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.375644 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.375643 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.375684 4979 server_base.cc:1034] running on GCE node
W20260619 02:50:11.375643 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.376686 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.377063 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.378443 4979 hybrid_clock.cc:648] HybridClock initialized: now 1781837411378427 us; error 41 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.381906 4979 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:50:11.382534 4979 webserver.cc:460] Webserver started at http://127.26.98.2:41663/ using document root <none> and password file <none>
I20260619 02:50:11.382802 4979 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.382860 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.383045 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.384402 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "016c7e0178074d49a9491773b5bb1050"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "8ff4cf031910cedfb9b729fbac802dae"
server_key_iv: "a34c9db294921ee4df4169e5a09a0d41"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.384810 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "016c7e0178074d49a9491773b5bb1050"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "8ff4cf031910cedfb9b729fbac802dae"
server_key_iv: "a34c9db294921ee4df4169e5a09a0d41"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.386605 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 02:50:11.388072 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.388420 4979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:11.388533 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "016c7e0178074d49a9491773b5bb1050"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "8ff4cf031910cedfb9b729fbac802dae"
server_key_iv: "a34c9db294921ee4df4169e5a09a0d41"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.388645 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.428484 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.431192 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.431347 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.431650 4979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:11.432035 4979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:11.432071 4979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.432097 4979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:11.432109 4979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.439450 4979 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:34559
I20260619 02:50:11.439503 5107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:34559 every 8 connection(s)
I20260619 02:50:11.439898 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:11.440315 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 4979
I20260619 02:50:11.440428 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:11.440667 27016 external_mini_cluster.cc:1428] Setting key a5dee529333ae4f5939d03d186aa0784
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
I20260619 02:50:11.455082 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.440819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:49441 (local address 127.26.98.62:45339)
0619 02:50:11.440932 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.440935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.441180 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:50:11.441788 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.441790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.441802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.441863 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.443383 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.444192 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.447024 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.447377 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.447488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.450227 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.450230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.450231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.450243 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.453106 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.453838 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.453841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.453842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.453929 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.454225 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.454225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.454226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.454378 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.454445 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.454779 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.454963 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:50:11.455459 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.440783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45339 (local address 127.26.98.2:49441)
0619 02:50:11.440987 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.441037 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.441691 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.442088 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.442094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.442796 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:11.443222 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.443229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.445809 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.445811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.446854 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.446858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.447530 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.447531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.447696 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.448410 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:11.448421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:11.450078 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:11.453257 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.453272 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.453284 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.453705 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.454006 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.454007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.454007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.454159 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.454438 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:11.454453 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:11.454630 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.454789 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.455128 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260619 02:50:11.456099 5108 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45339
I20260619 02:50:11.456235 5108 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:11.456493 5108 heartbeater.cc:511] Master 127.26.98.62:45339 requested a full tablet report, sending...
I20260619 02:50:11.457005 4783 ts_manager.cc:194] Registered new tserver with Master: 016c7e0178074d49a9491773b5bb1050 (127.26.98.2:34559)
I20260619 02:50:11.457576 4783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:49441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:50:11.479566 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.580366 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.580662 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.580693 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.582129 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.582187 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.582221 5113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:50:11.584028 5113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:50:11.584499 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.584937 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.591398 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.591372 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.591365 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.591702 5113 server_base.cc:1034] running on GCE node
I20260619 02:50:11.591948 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.592247 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.593422 5113 hybrid_clock.cc:648] HybridClock initialized: now 1781837411593375 us; error 68 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.596925 5113 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:50:11.597568 5113 webserver.cc:460] Webserver started at http://127.26.98.3:45787/ using document root <none> and password file <none>
I20260619 02:50:11.597951 5113 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.598064 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.598222 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.599622 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "f2372325ce284c70a5c18670d73578a8"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "6432cfdf341fc351820666c71d4e188b"
server_key_iv: "9efbea95333d05e59994c00bd13cf6e4"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.600059 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "f2372325ce284c70a5c18670d73578a8"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "6432cfdf341fc351820666c71d4e188b"
server_key_iv: "9efbea95333d05e59994c00bd13cf6e4"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.601706 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:11.602831 5128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.603134 5113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:11.603286 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "f2372325ce284c70a5c18670d73578a8"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-0f9d"
server_key: "6432cfdf341fc351820666c71d4e188b"
server_key_iv: "9efbea95333d05e59994c00bd13cf6e4"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.603395 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.623170 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.626209 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.626402 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.626739 5113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:11.627161 5113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:11.627200 5113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.627235 5113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:11.627255 5113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.636368 5113 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:42203
I20260619 02:50:11.636492 5241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:42203 every 8 connection(s)
I20260619 02:50:11.636920 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[4738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:50:11.647193 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 5113
I20260619 02:50:11.647334 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:11.647579 27016 external_mini_cluster.cc:1428] Setting key 4e18e5f51e35e97ba82c4ced376432a1
I20260619 02:50:11.650619 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.637964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:57021 (local address 127.26.98.62:45339)
0619 02:50:11.638040 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.638042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.638214 (+ 172us) server_negotiation.cc:407] Connection header received
0619 02:50:11.638777 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.638779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.638792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.638866 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.640217 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.640906 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.642275 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.642561 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.642642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.646141 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.646144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.646144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.646156 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.647848 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.648821 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.648822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.648823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.648913 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.649446 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.649448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.649449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.649667 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.649783 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.650351 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.650485 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:50:11.650780 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.637814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45339 (local address 127.26.98.3:57021)
0619 02:50:11.638014 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.638068 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.638670 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.638995 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.639000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.639705 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:11.639992 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.639998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.641106 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.641107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.642123 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.642127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.642679 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.642680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.642832 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.643912 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:11.643926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:11.645955 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:11.647959 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.648140 (+ 181us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.648160 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.648702 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.649043 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.649044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.649044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.649228 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.649859 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:11.649868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:11.650079 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.650278 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.650519 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260619 02:50:11.651376 5242 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45339
I20260619 02:50:11.651491 5242 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:11.651744 5242 heartbeater.cc:511] Master 127.26.98.62:45339 requested a full tablet report, sending...
I20260619 02:50:11.652204 4783 ts_manager.cc:194] Registered new tserver with Master: f2372325ce284c70a5c18670d73578a8 (127.26.98.3:42203)
I20260619 02:50:11.652628 4783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:57021
I20260619 02:50:11.660410 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:50:11.669452 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.662023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.26.98.62:45339)
0619 02:50:11.662173 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.662175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.662240 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:50:11.662330 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.662333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.662345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.662394 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.663087 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.663648 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.664434 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.664643 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.664703 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.666066 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.666067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.666068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.666078 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.667719 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.668494 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.668496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.668496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.668589 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.668792 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.668793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.668794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.668978 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.669077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.669202 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.669348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 02:50:11.672114 4783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:11.674196 4783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:11.682698 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:34559 (local address 127.0.0.1:58816)
0619 02:50:11.678988 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.678998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.679054 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.679241 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.679242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.679244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.679299 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.679300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680200 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.680202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.681396 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.681399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.682339 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.682340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.682419 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.682436 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.682510 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.682579 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":123,"thread_start_us":78,"threads_started":1}
I20260619 02:50:11.682813 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:42203 (local address 127.0.0.1:60832)
0619 02:50:11.678296 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.678306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.678360 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.678741 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.678742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.678744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.678787 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.678788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680205 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.680206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.681661 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.681663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.682577 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.682578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.682635 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.682638 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.682714 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.682757 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":60,"thread_start_us":91,"threads_started":1}
I20260619 02:50:11.683110 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.26.98.3:42203)
0619 02:50:11.678246 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.678252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.678409 (+ 157us) server_negotiation.cc:407] Connection header received
0619 02:50:11.678461 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.678464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.678480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.678753 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.678835 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680077 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.681748 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.682481 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.682552 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.682607 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.682936 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.683005 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":113,"threads_started":1}
I20260619 02:50:11.683369 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.26.98.2:34559)
0619 02:50:11.678116 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.678120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.679065 (+ 945us) server_negotiation.cc:407] Connection header received
0619 02:50:11.679148 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.679151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.679167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.679240 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.679350 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680053 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.681500 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.682232 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.682317 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.682392 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.682540 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.683267 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260619 02:50:11.684124 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:40101 (local address 127.0.0.1:40762)
0619 02:50:11.677974 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.678024 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.678111 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.678397 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.678401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.678404 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.678501 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.678503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680256 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.680257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.681655 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.681658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.683808 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.683809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.683917 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.683922 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.683969 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.684025 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":106,"thread_start_us":80,"threads_started":1}
I20260619 02:50:11.684118 5042 tablet_service.cc:1467] Processing CreateTablet for tablet bb6968f1c12b438d89da2552bcd90b62 (DEFAULT_TABLE table=test-table [id=f721b433bc6a4c5a83e6474f4f877747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:11.684365 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.677746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.26.98.1:40101)
0619 02:50:11.677983 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.677989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.678207 (+ 218us) server_negotiation.cc:407] Connection header received
0619 02:50:11.678263 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.678266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.678286 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.678370 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.678658 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.680154 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.682655 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.683639 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.683711 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.683770 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.684228 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.684283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":67,"threads_started":1}
I20260619 02:50:11.684571 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6968f1c12b438d89da2552bcd90b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:11.684631 5176 tablet_service.cc:1467] Processing CreateTablet for tablet bb6968f1c12b438d89da2552bcd90b62 (DEFAULT_TABLE table=test-table [id=f721b433bc6a4c5a83e6474f4f877747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:11.684973 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6968f1c12b438d89da2552bcd90b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:11.688062 4908 tablet_service.cc:1467] Processing CreateTablet for tablet bb6968f1c12b438d89da2552bcd90b62 (DEFAULT_TABLE table=test-table [id=f721b433bc6a4c5a83e6474f4f877747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:11.688231 5259 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Bootstrap starting.
I20260619 02:50:11.688532 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6968f1c12b438d89da2552bcd90b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:11.689126 5259 tablet_bootstrap.cc:654] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:11.689828 5259 log.cc:824] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:11.690778 5259 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: No bootstrap required, opened a new log
I20260619 02:50:11.690914 5259 ts_tablet_manager.cc:1397] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:50:11.691671 5261 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Bootstrap starting.
I20260619 02:50:11.692569 5261 tablet_bootstrap.cc:654] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:11.692548 5262 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Bootstrap starting.
I20260619 02:50:11.693341 5262 tablet_bootstrap.cc:654] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:11.693416 5261 log.cc:824] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:11.693620 5259 raft_consensus.cc:348] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.693825 5259 raft_consensus.cc:374] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:11.693851 5259 raft_consensus.cc:729] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016c7e0178074d49a9491773b5bb1050, State: Initialized, Role: FOLLOWER
I20260619 02:50:11.693986 5259 consensus_queue.cc:260] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.694103 5262 log.cc:824] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:11.694322 5259 ts_tablet_manager.cc:1428] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:50:11.694464 5108 heartbeater.cc:503] Master 127.26.98.62:45339 was elected leader, sending a full tablet report...
I20260619 02:50:11.695529 5262 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: No bootstrap required, opened a new log
I20260619 02:50:11.695616 5262 ts_tablet_manager.cc:1397] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:50:11.695839 5261 tablet_bootstrap.cc:492] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: No bootstrap required, opened a new log
I20260619 02:50:11.695955 5261 ts_tablet_manager.cc:1397] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:50:11.697383 5262 raft_consensus.cc:348] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.697597 5262 raft_consensus.cc:374] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:11.697636 5262 raft_consensus.cc:729] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983d893ab4db42468308351d93e7f0b2, State: Initialized, Role: FOLLOWER
I20260619 02:50:11.697785 5262 consensus_queue.cc:260] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.697930 5261 raft_consensus.cc:348] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.698151 5261 raft_consensus.cc:374] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:11.698158 5262 ts_tablet_manager.cc:1428] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:50:11.698200 5261 raft_consensus.cc:729] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2372325ce284c70a5c18670d73578a8, State: Initialized, Role: FOLLOWER
I20260619 02:50:11.698207 4974 heartbeater.cc:503] Master 127.26.98.62:45339 was elected leader, sending a full tablet report...
I20260619 02:50:11.698355 5261 consensus_queue.cc:260] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.698726 5261 ts_tablet_manager.cc:1428] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:50:11.698786 5242 heartbeater.cc:503] Master 127.26.98.62:45339 was elected leader, sending a full tablet report...
W20260619 02:50:11.717674 4975 tablet.cc:2369] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:11.767557 5264 raft_consensus.cc:482] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:11.767758 5264 raft_consensus.cc:504] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.768402 5264 leader_election.cc:288] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983d893ab4db42468308351d93e7f0b2 (127.26.98.1:40101), f2372325ce284c70a5c18670d73578a8 (127.26.98.3:42203)
I20260619 02:50:11.772111 5267 raft_consensus.cc:482] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:11.772389 5267 raft_consensus.cc:504] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.772544 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.768687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:40101 (local address 127.26.98.2:50487)
0619 02:50:11.768808 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.768822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.768882 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.769296 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.769298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.769301 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.769417 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.769419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.770294 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.770295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.771507 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.771511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.772218 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.772219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.772323 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.772325 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.772377 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.772433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":34}
I20260619 02:50:11.772544 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.768837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:35493 (local address 127.26.98.3:42203)
0619 02:50:11.768904 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.768906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.769139 (+ 233us) server_negotiation.cc:407] Connection header received
0619 02:50:11.769289 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.769290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.769308 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.769375 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.769747 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.770304 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.771626 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.772127 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.772206 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.772220 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.772427 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.772481 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:50:11.772573 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.768863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:50487 (local address 127.26.98.1:40101)
0619 02:50:11.768909 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.768911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.768920 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:11.768961 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.768962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.769034 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.769247 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.769561 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.770170 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.771638 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.772104 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.772197 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.772214 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.772433 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.772493 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:50:11.772810 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.768687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:42203 (local address 127.26.98.2:35493)
0619 02:50:11.769006 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.769019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.769212 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.769461 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.769463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.769469 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.769591 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.769593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.770422 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.770423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.771502 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.771506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.772248 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.772249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.772318 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.772323 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.772654 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.772723 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":71,"thread_start_us":139,"threads_started":1}
I20260619 02:50:11.773011 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "016c7e0178074d49a9491773b5bb1050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d893ab4db42468308351d93e7f0b2" is_pre_election: true
I20260619 02:50:11.773142 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "016c7e0178074d49a9491773b5bb1050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2372325ce284c70a5c18670d73578a8" is_pre_election: true
I20260619 02:50:11.773206 4928 raft_consensus.cc:2454] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016c7e0178074d49a9491773b5bb1050 in term 0.
I20260619 02:50:11.773241 5196 raft_consensus.cc:2454] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016c7e0178074d49a9491773b5bb1050 in term 0.
I20260619 02:50:11.773767 4996 leader_election.cc:302] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016c7e0178074d49a9491773b5bb1050, f2372325ce284c70a5c18670d73578a8; no voters:
I20260619 02:50:11.773959 5264 raft_consensus.cc:2781] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:11.774032 5264 raft_consensus.cc:482] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:11.774062 5264 raft_consensus.cc:3037] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:11.775252 5264 raft_consensus.cc:504] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.775434 5264 leader_election.cc:288] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [CANDIDATE]: Term 1 election: Requested vote from peers 983d893ab4db42468308351d93e7f0b2 (127.26.98.1:40101), f2372325ce284c70a5c18670d73578a8 (127.26.98.3:42203)
I20260619 02:50:11.775650 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "016c7e0178074d49a9491773b5bb1050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2372325ce284c70a5c18670d73578a8"
I20260619 02:50:11.775750 5196 raft_consensus.cc:3037] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:11.776672 5196 raft_consensus.cc:2454] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016c7e0178074d49a9491773b5bb1050 in term 1.
I20260619 02:50:11.776834 5267 leader_election.cc:288] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983d893ab4db42468308351d93e7f0b2 (127.26.98.1:40101), 016c7e0178074d49a9491773b5bb1050 (127.26.98.2:34559)
I20260619 02:50:11.776883 4996 leader_election.cc:302] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016c7e0178074d49a9491773b5bb1050, f2372325ce284c70a5c18670d73578a8; no voters:
I20260619 02:50:11.776964 5264 raft_consensus.cc:2781] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:11.777251 5264 raft_consensus.cc:686] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 1 LEADER]: Becoming Leader. State: Replica: 016c7e0178074d49a9491773b5bb1050, State: Running, Role: LEADER
I20260619 02:50:11.776932 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "016c7e0178074d49a9491773b5bb1050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d893ab4db42468308351d93e7f0b2"
I20260619 02:50:11.777387 5264 consensus_queue.cc:237] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } }
I20260619 02:50:11.777462 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.773502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:40101 (local address 127.26.98.3:51117)
0619 02:50:11.773702 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.773711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.773786 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.774011 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.774012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.774014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.774227 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.774230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.775083 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.775084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.776206 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.776208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.776938 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.776940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.777111 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.777117 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.777322 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.777372 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":141,"threads_started":1}
I20260619 02:50:11.777633 4928 raft_consensus.cc:3037] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:11.778067 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.773538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:51117 (local address 127.26.98.1:40101)
0619 02:50:11.773603 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.773605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.773764 (+ 159us) server_negotiation.cc:407] Connection header received
0619 02:50:11.773847 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.773848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.773858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.773884 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.774314 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.774986 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.776281 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.776723 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.776793 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.776807 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.777233 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.777301 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:50:11.778836 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.773432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:34559 (local address 127.26.98.3:33971)
0619 02:50:11.773531 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.773542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.773582 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.773779 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.773780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.773783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:11.773852 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.773854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.774858 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.774861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.776198 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.776202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.778518 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.778520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.778633 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.778638 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.778690 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.778749 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 02:50:11.779223 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.773490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:33971 (local address 127.26.98.2:34559)
0619 02:50:11.773535 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.773537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.773567 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:50:11.773649 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.773650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.773663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.773726 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:11.773919 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.774675 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.776850 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.778274 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.778443 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.778459 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.779091 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.779153 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:11.779274 4928 raft_consensus.cc:2454] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016c7e0178074d49a9491773b5bb1050 in term 1.
I20260619 02:50:11.779256 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "f2372325ce284c70a5c18670d73578a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d893ab4db42468308351d93e7f0b2" is_pre_election: true
I20260619 02:50:11.779414 4927 raft_consensus.cc:2379] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2372325ce284c70a5c18670d73578a8 in current term 1: Already voted for candidate 016c7e0178074d49a9491773b5bb1050 in this term.
I20260619 02:50:11.779409 4783 catalog_manager.cc:5495] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 reported cstate change: term changed from 0 to 1, leader changed from <none> to 016c7e0178074d49a9491773b5bb1050 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "016c7e0178074d49a9491773b5bb1050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016c7e0178074d49a9491773b5bb1050" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34559 } health_report { overall_health: HEALTHY } } }
I20260619 02:50:11.779773 5062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6968f1c12b438d89da2552bcd90b62" candidate_uuid: "f2372325ce284c70a5c18670d73578a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016c7e0178074d49a9491773b5bb1050" is_pre_election: true
I20260619 02:50:11.780097 5130 leader_election.cc:302] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2372325ce284c70a5c18670d73578a8; no voters: 016c7e0178074d49a9491773b5bb1050, 983d893ab4db42468308351d93e7f0b2
I20260619 02:50:11.780365 5267 raft_consensus.cc:2726] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 02:50:11.804121 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.800931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.26.98.2:34559)
0619 02:50:11.800997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.800999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.801010 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:11.801051 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.801052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.801072 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.801157 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:11.801384 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.801977 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.803051 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.803284 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.803364 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.803612 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:11.803764 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:11.803859 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.803964 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.803999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:50:11.806779 4927 raft_consensus.cc:1261] T bb6968f1c12b438d89da2552bcd90b62 P 983d893ab4db42468308351d93e7f0b2 [term 1 FOLLOWER]: Refusing update from remote peer 016c7e0178074d49a9491773b5bb1050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:11.806839 5196 raft_consensus.cc:1261] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 1 FOLLOWER]: Refusing update from remote peer 016c7e0178074d49a9491773b5bb1050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:11.807281 5270 consensus_queue.cc:1035] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983d893ab4db42468308351d93e7f0b2" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 40101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:11.807538 5264 consensus_queue.cc:1035] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2372325ce284c70a5c18670d73578a8" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:11.810385 5273 mvcc.cc:205] Tried to move back new op lower bound from 7298406038756749312 to 7298406038641074176. Current Snapshot: MvccSnapshot[applied={T|T < 7298406038756749312}]
I20260619 02:50:11.811087 5275 mvcc.cc:205] Tried to move back new op lower bound from 7298406038756749312 to 7298406038641074176. Current Snapshot: MvccSnapshot[applied={T|T < 7298406038756749312 or (T in {7298406038756749312})}]
I20260619 02:50:11.814071 4783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:11.814222 4783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:11.815536 4783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87e833479beb4b7fbd1bb7bce1b43128: Sending DeleteTablet for 3 replicas of tablet bb6968f1c12b438d89da2552bcd90b62
I20260619 02:50:11.816151 5042 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6968f1c12b438d89da2552bcd90b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:58816
I20260619 02:50:11.816170 4908 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6968f1c12b438d89da2552bcd90b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:40762
I20260619 02:50:11.816532 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6968f1c12b438d89da2552bcd90b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:60832
I20260619 02:50:11.816890 5284 tablet_replica.cc:330] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: stopping tablet replica
I20260619 02:50:11.816938 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4845
I20260619 02:50:11.817108 5284 raft_consensus.cc:2229] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:11.817277 5284 raft_consensus.cc:2258] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:11.817958 5284 ts_tablet_manager.cc:1905] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:11.820178 5284 ts_tablet_manager.cc:1918] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:11.820299 5284 log.cc:1196] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/bb6968f1c12b438d89da2552bcd90b62
I20260619 02:50:11.820608 5284 ts_tablet_manager.cc:1939] T bb6968f1c12b438d89da2552bcd90b62 P f2372325ce284c70a5c18670d73578a8: Deleting consensus metadata
I20260619 02:50:11.821305 4768 catalog_manager.cc:4841] TS f2372325ce284c70a5c18670d73578a8 (127.26.98.3:42203): tablet bb6968f1c12b438d89da2552bcd90b62 (table test-table [id=f721b433bc6a4c5a83e6474f4f877747]) successfully deleted
I20260619 02:50:11.824105 5283 tablet_replica.cc:330] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: stopping tablet replica
I20260619 02:50:11.824299 5283 raft_consensus.cc:2229] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:11.824465 5283 raft_consensus.cc:2258] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:50:11.824820 4767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:40101 (error 108)
I20260619 02:50:11.825182 5283 ts_tablet_manager.cc:1905] T bb6968f1c12b438d89da2552bcd90b62 P 016c7e0178074d49a9491773b5bb1050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:11.825418 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.825287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:40101 (local address 127.0.0.1:40766)
0619 02:50:11.825345 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.825381 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:50:11.825536 4767 catalog_manager.cc:4584] TS 983d893ab4db42468308351d93e7f0b2 (127.26.98.1:40101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb6968f1c12b438d89da2552bcd90b62: Network error: Client connection negotiation failed: client connection to 127.26.98.1:40101: connect: Connection refused (error 111)
I20260619 02:50:11.825811 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4979
I20260619 02:50:11.831284 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 5113
I20260619 02:50:11.831466 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.831273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:34559 (local address 127.0.0.1:58826)
0619 02:50:11.831352 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.831400 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:50:11.836311 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 4751
2026-06-19T02:50:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2252 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[5288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[5288](info): set up 2 sockets
Jun 19 02:50:11 dist-test-slave-0f9d krb5kdc[5288](info): commencing operation
krb5kdc: starting...
W20260619 02:50:12.840524 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:50:12 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangeradmin_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangeradmin_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangerlookup_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangerlookup_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
I20260619 02:50:13.052134 27016 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcBfLLw/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/postgres -l logfile start
2026-06-19 02:50:16.282 UTC [5318] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:50:16.282 UTC [5318] LOG: listening on IPv4 address "127.26.98.20", port 37419
2026-06-19 02:50:16.286 UTC [5318] LOG: listening on Unix socket "/tmp/.s.PGSQL.37419"
2026-06-19 02:50:16.311 UTC [5321] LOG: database system was shut down at 2026-06-19 02:50:14 UTC
2026-06-19 02:50:16.316 UTC [5318] LOG: database system is ready to accept connections
I20260619 02:50:17.220213 27016 mini_postgres.cc:96] Postgres bound to 37419
2026-06-19 02:50:17.228 UTC [5329] FATAL: database "slave" does not exist
127.26.98.20:37419 - accepting connections
I20260619 02:50:17.229784 27016 mini_ranger.cc:161] Starting Ranger...
I20260619 02:50:17.250341 27016 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:50:17.849763 27016 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:50:17.849869 27016 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin
2026-06-19 02:50:18,189 [I] DB FLAVOR :POSTGRES
2026-06-19 02:50:18,190 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:50:18,190 [I] Checking connection..
2026-06-19 02:50:18,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:50:18,516 [I] Checking connection passed.
2026-06-19 02:50:18,517 [I] --------- Verifying version history table ---------
2026-06-19 02:50:18,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:18,820 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:50:18,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:19,147 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:50:19,148 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:50:19,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:50:19.460 UTC [5393] WARNING: there is no transaction in progress
2026-06-19 02:50:19,470 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:50:19,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:19,776 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:50:19,776 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:50:19,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:50:20,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:50:20,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0f9d.c.gcp-upstream.internal','N') ;"
2026-06-19 02:50:20,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:50:21,001 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:50:21,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:50:21,332 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:50:21,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:50:21,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:50:21,950 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:50:21,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:50:23.317 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.339 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.361 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.385 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.408 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.432 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.512 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.523 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.541 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.555 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.568 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.587 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.600 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.610 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.620 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:23.630 UTC [5501] WARNING: there is no transaction in progress
2026-06-19 02:50:24,635 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:50:24,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:50:24,944 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:50:24,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:50:25,287 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:50:25,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:50:25,621 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:50:25,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:50:25,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:50:26,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0f9d.c.gcp-upstream.internal';"
2026-06-19 02:50:26,607 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:50:26,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:50:26,931 [I] DB_PATCHES have already been applied
I20260619 02:50:26.939009 27016 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcBfLLw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:50:26.939106 27016 mini_ranger.cc:193] Using host: 127.26.98.20
I20260619 02:50:26.943698 27016 mini_ranger.cc:227] Ranger admin URL: http://127.26.98.20:34895
Jun 19, 2026 2:50:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:50:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:50:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:50:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:50:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:50:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:50:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:50:29.112546 27016 mini_ranger.cc:160] Time spent starting Ranger: real 11.883s user 0.000s sys 0.015s
Jun 19, 2026 2:50:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:50:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:50:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:50:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:50:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:50:58.409279 27016 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangerkms_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/rangerkms_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.26.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:50:58 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:50:58.521872 27016 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:50:58.535161 27016 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:50:59.586032 27016 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:50:59.586151 27016 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms
I20260619 02:50:59.586167 27016 mini_ranger_kms.cc:227] Using postgres at 127.26.98.20:37419
2026-06-19 02:50:59,795 [I] DB FLAVOR :POSTGRES
2026-06-19 02:50:59,796 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:50:59,796 [I] Checking connection
2026-06-19 02:50:59,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:51:00,143 [I] connection success
2026-06-19 02:51:00,144 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:51:00,144 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:51:00,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:51:00,480 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:51:00,480 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:00,481 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:51:00,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:37419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:51:00,848 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:51:01.512861 27016 mini_ranger_kms.cc:310] Created kms service
I20260619 02:51:01.798197 27016 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:51:01.860275 27016 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:51:01 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:02.477345 27016 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:51:02.477500 27016 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:51:02.477519 27016 mini_ranger_kms.cc:240] Using host: 127.26.98.20
I20260619 02:51:02.482086 27016 mini_ranger_kms.cc:276] Ranger KMS PID: 5683
I20260619 02:51:02.482282 27016 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.26.98.20:38241
Jun 19, 2026 2:51:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:51:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:51:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:51:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:51:04.653923 27016 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.132s user 0.008s sys 0.183s
I20260619 02:51:04.654059 27016 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:51:04.654166 27016 mini_ranger_kms.cc:401] 127.26.98.20:38241/kms/v1/keys
Jun 19 02:51:05 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:51:05 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
Jun 19 02:51:06 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
Jun 19 02:51:09 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:51:09.338970 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36351
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36351
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:38241/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:09.463522 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:09.464021 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:09.464090 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:09.466864 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:51:09.467003 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:09.467024 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:09.467041 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:51:09.467060 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:51:09.470304 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal
--ranger_kms_url=127.26.98.20:38241/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36351
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:51:09.470700 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:09.471027 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:09.477463 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:09.477463 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:09.477466 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:09.477986 5733 server_base.cc:1034] running on GCE node
I20260619 02:51:09.478343 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:09.478804 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:09.480012 5733 hybrid_clock.cc:648] HybridClock initialized: now 1781837469479997 us; error 51 us; skew 500 ppm
Jun 19 02:51:09 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837469, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:09.484094 5733 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:51:09.485011 5733 webserver.cc:460] Webserver started at http://127.26.98.62:40609/ using document root <none> and password file <none>
I20260619 02:51:09.485352 5733 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:09.485426 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:09.485620 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:09 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837469, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:09.555827 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "7d03ea4d94d945cbb926e55d0544948b"
format_stamp: "Formatted at 2026-06-19 02:51:09 on dist-test-slave-0f9d"
server_key: "3428344cba44be911eac0756045c3cdc"
server_key_iv: "2229a92494ec680ce2a774bf3b19f5d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:09.556417 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "7d03ea4d94d945cbb926e55d0544948b"
format_stamp: "Formatted at 2026-06-19 02:51:09 on dist-test-slave-0f9d"
server_key: "3428344cba44be911eac0756045c3cdc"
server_key_iv: "2229a92494ec680ce2a774bf3b19f5d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:09.558924 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:51:09.581475 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:09.581894 5733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:51:09.582080 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "7d03ea4d94d945cbb926e55d0544948b"
format_stamp: "Formatted at 2026-06-19 02:51:09 on dist-test-slave-0f9d"
server_key: "3428344cba44be911eac0756045c3cdc"
server_key_iv: "2229a92494ec680ce2a774bf3b19f5d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:09.582223 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:09.597295 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:09.600526 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:09.600769 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:09.607929 5733 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36351
I20260619 02:51:09.607985 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36351 every 8 connection(s)
I20260619 02:51:09.608560 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:51:09.609576 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:09.609582 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 5733
I20260619 02:51:09.609691 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:51:09.609916 27016 external_mini_cluster.cc:1428] Setting key 1e021e66906e94bb34862d7c2e7616f6
I20260619 02:51:09.614225 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: Bootstrap starting.
I20260619 02:51:09.615314 5801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:09.615724 5801 log.cc:824] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:09.616797 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: No bootstrap required, opened a new log
Jun 19 02:51:09 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:51:09.619890 5801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } }
I20260619 02:51:09.620142 5801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:09.620211 5801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d03ea4d94d945cbb926e55d0544948b, State: Initialized, Role: FOLLOWER
I20260619 02:51:09.620424 5801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } }
I20260619 02:51:09.620518 5801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:51:09.620556 5801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:51:09.620626 5801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:09.621511 5801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } }
I20260619 02:51:09.621675 5801 leader_election.cc:302] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d03ea4d94d945cbb926e55d0544948b; no voters:
I20260619 02:51:09.621964 5801 leader_election.cc:288] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:51:09.622013 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:51:09.622246 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [term 1 LEADER]: Becoming Leader. State: Replica: 7d03ea4d94d945cbb926e55d0544948b, State: Running, Role: LEADER
I20260619 02:51:09.622416 5801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:51:09.622392 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } }
I20260619 02:51:09.622982 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d03ea4d94d945cbb926e55d0544948b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } } }
I20260619 02:51:09.623077 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:09.623355 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d03ea4d94d945cbb926e55d0544948b. Latest consensus state: current_term: 1 leader_uuid: "7d03ea4d94d945cbb926e55d0544948b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d03ea4d94d945cbb926e55d0544948b" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36351 } } }
I20260619 02:51:09.623423 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:09.624554 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:51:09.624693 5801 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:51:09.625276 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:51:09.629354 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:09.610630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.26.98.62:36351)
0619 02:51:09.610865 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:09.610910 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:09.610956 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:51:09.611599 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:09.611612 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:09.611698 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:09.611897 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:09.612828 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:09.614414 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:09.615671 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:09.616481 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:09.616641 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:09.620292 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:09.620299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:09.620299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:09.620318 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:09.623160 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:09.623869 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:09.623874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:09.623875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:09.624124 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:09.624444 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:09.624445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:09.624446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:09.624825 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:09.625538 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:09.625626 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 02:51:09.628739 (+ 3113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260619 02:51:09.635890 5810 catalog_manager.cc:1347] Generated new cluster ID: bbb911094f6f4154a011d39e2295c13d
I20260619 02:51:09.636000 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:51:09.663906 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:51:09.664804 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:51:09.672901 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: Generated new TSK 0
I20260619 02:51:09.673342 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:51:10.602982 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:38241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:10.735950 5846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:10.736238 5846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:10.736351 5846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:10.738456 5846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:10.738535 5846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:10.738586 5846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:51:10.740885 5846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.26.98.20:38241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:51:10.741250 5846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:10.741551 5846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:10.747792 5854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:10.747925 5846 server_base.cc:1034] running on GCE node
W20260619 02:51:10.747822 5852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:10.748003 5851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:10.748279 5846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:10.748605 5846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:10.749787 5846 hybrid_clock.cc:648] HybridClock initialized: now 1781837470749758 us; error 57 us; skew 500 ppm
Jun 19 02:51:10 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837470, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:10.753921 5846 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:51:10.754778 5846 webserver.cc:460] Webserver started at http://127.26.98.1:39761/ using document root <none> and password file <none>
I20260619 02:51:10.755159 5846 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:10.755259 5846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:10.755414 5846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:10 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837470, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:10.775221 5846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "82035dbbf7144ab59943bf402b0e272d"
format_stamp: "Formatted at 2026-06-19 02:51:10 on dist-test-slave-0f9d"
server_key: "26e01abfc46b2fc3d3a8aa0536aa99a0"
server_key_iv: "cbf778afe9ded22a0e0e6bb764ae29fc"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:10.776208 5846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "82035dbbf7144ab59943bf402b0e272d"
format_stamp: "Formatted at 2026-06-19 02:51:10 on dist-test-slave-0f9d"
server_key: "26e01abfc46b2fc3d3a8aa0536aa99a0"
server_key_iv: "cbf778afe9ded22a0e0e6bb764ae29fc"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:10.779075 5846 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:51:10.807109 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:10.807616 5846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 02:51:10.807780 5846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "82035dbbf7144ab59943bf402b0e272d"
format_stamp: "Formatted at 2026-06-19 02:51:10 on dist-test-slave-0f9d"
server_key: "26e01abfc46b2fc3d3a8aa0536aa99a0"
server_key_iv: "cbf778afe9ded22a0e0e6bb764ae29fc"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:10.807924 5846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:10.837359 5846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:10.840153 5846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:10.840349 5846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:10.840651 5846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:10.841076 5846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:10.841115 5846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:10.841231 5846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:10.841255 5846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:10.849611 5846 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:33733
I20260619 02:51:10.849730 5976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:33733 every 8 connection(s)
I20260619 02:51:10.850118 5846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:51:10.851495 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 5846
I20260619 02:51:10.851624 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:51:10.851840 27016 external_mini_cluster.cc:1428] Setting key 0cca3095ee4105e9f982802f1c80b38a
Jun 19 02:51:10 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837470, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:51:10.864678 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:10.851106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36351 (local address 127.26.98.1:47525)
0619 02:51:10.851423 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:10.851482 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:10.852278 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:10.852724 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:10.852730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:10.853038 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:10.853380 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:10.853387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:10.854516 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:10.854517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:10.856044 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:10.856049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:10.856716 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:10.856717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:10.856934 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:10.857652 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:10.857666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:10.859796 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:10.862352 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:10.862369 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:10.862374 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:10.862823 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:10.863121 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:10.863122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:10.863122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:10.863294 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:10.863646 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:10.863655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:10.863914 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 02:51:10.864036 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:51:10.864253 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
I20260619 02:51:10.875016 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:10.851121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:47525 (local address 127.26.98.62:36351)
0619 02:51:10.851283 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:10.851286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:10.851676 (+ 390us) server_negotiation.cc:407] Connection header received
0619 02:51:10.852462 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:10.852463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:10.852479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:10.852567 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:10.853507 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:10.854369 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:10.856196 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:10.856541 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:10.856619 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:10.859990 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:10.859993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:10.859993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:10.860007 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:10.862196 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:10.862966 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:10.862968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:10.862969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:10.863050 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:10.863365 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:10.863366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:10.863367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:10.863541 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:10.863617 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:10.864722 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0619 02:51:10.874725 (+ 10003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260619 02:51:10.876000 5977 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36351
I20260619 02:51:10.876163 5977 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:10.876453 5977 heartbeater.cc:511] Master 127.26.98.62:36351 requested a full tablet report, sending...
I20260619 02:51:10.877561 5765 ts_manager.cc:194] Registered new tserver with Master: 82035dbbf7144ab59943bf402b0e272d (127.26.98.1:33733)
I20260619 02:51:10.878602 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:47525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:51:10.905403 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:38241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:51:10 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837470, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:51:11.050132 5985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:11.050544 5985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:11.050592 5985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:11.053081 5985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:11.053270 5985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:11.053341 5985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:51:11.056396 5985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.26.98.20:38241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:51:11.057008 5985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:11.057430 5985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:11.066220 5993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:11.066304 5990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:11.066654 5985 server_base.cc:1034] running on GCE node
W20260619 02:51:11.066763 5991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:11.067096 5985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:11.067543 5985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:11.068778 5985 hybrid_clock.cc:648] HybridClock initialized: now 1781837471068740 us; error 56 us; skew 500 ppm
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:11.072580 5985 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:51:11.073266 5985 webserver.cc:460] Webserver started at http://127.26.98.2:45525/ using document root <none> and password file <none>
I20260619 02:51:11.073565 5985 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:11.073623 5985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:11.073787 5985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:11.098810 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "9243529888d84499a730b76f17a92a51"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "2ed3d553bf2ea78277553a73f39198a6"
server_key_iv: "f2b0775b202650118e5add9dca3c1641"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.099611 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "9243529888d84499a730b76f17a92a51"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "2ed3d553bf2ea78277553a73f39198a6"
server_key_iv: "f2b0775b202650118e5add9dca3c1641"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.101492 5985 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:51:11.123106 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.123349 5985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:51:11.123476 5985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "9243529888d84499a730b76f17a92a51"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "2ed3d553bf2ea78277553a73f39198a6"
server_key_iv: "f2b0775b202650118e5add9dca3c1641"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.123606 5985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:11.139396 5985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:11.142481 5985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:11.142802 5985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:11.143210 5985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:11.143863 5985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:11.143946 5985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.144009 5985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:11.144052 5985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.153213 5985 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:33649
I20260619 02:51:11.154052 5985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:51:11.157756 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 5985
I20260619 02:51:11.157912 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:51:11.158142 27016 external_mini_cluster.cc:1428] Setting key 04f9ff7995048da85d7f1059d9bbb28c
I20260619 02:51:11.167615 6114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:33649 every 8 connection(s)
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:51:11.183913 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.168624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:36547 (local address 127.26.98.62:36351)
0619 02:51:11.168692 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.168695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.169998 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 02:51:11.170977 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.170980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.170998 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.171034 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:11.172193 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.173058 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.174728 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.175103 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.175194 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.179369 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:11.179373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:11.179374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:11.179387 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:11.181126 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.181994 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.181997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.181997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.182208 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.182720 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.182723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.182723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.182981 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:11.183069 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.183600 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.183752 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
I20260619 02:51:11.184854 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.168638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36351 (local address 127.26.98.2:36547)
0619 02:51:11.169740 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.169831 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.170746 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.171305 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.171311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.171656 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:11.172039 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.172048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.173236 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.173237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.174581 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.174586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.175384 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.175389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.175646 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.177126 (+ 1480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:11.177142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:11.179184 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:11.181267 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:11.181289 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:11.181293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:11.181804 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:11.182343 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:11.182344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:11.182345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:11.182573 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:11.183076 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:11.183088 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:11.183377 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.184250 (+ 873us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.184516 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":138,"threads_started":1}
I20260619 02:51:11.185794 6115 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36351
I20260619 02:51:11.185954 6115 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:11.186234 6115 heartbeater.cc:511] Master 127.26.98.62:36351 requested a full tablet report, sending...
I20260619 02:51:11.186851 5765 ts_manager.cc:194] Registered new tserver with Master: 9243529888d84499a730b76f17a92a51 (127.26.98.2:33649)
I20260619 02:51:11.187476 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:36547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:51:11.211355 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:38241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:11.343359 6120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:11.343851 6120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:11.343972 6120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:11.346468 6120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:11.346654 6120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:11.346753 6120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:51:11.349720 6120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.26.98.20:38241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:51:11.350248 6120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:11.350553 6120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:11.356691 6126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:11.357373 6125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:11.356709 6128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:11.359138 6120 server_base.cc:1034] running on GCE node
I20260619 02:51:11.359478 6120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:11.359891 6120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:11.363204 6120 hybrid_clock.cc:648] HybridClock initialized: now 1781837471363179 us; error 53 us; skew 500 ppm
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:11.368175 6120 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:51:11.369064 6120 webserver.cc:460] Webserver started at http://127.26.98.3:36521/ using document root <none> and password file <none>
I20260619 02:51:11.369488 6120 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:11.369575 6120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:11.369735 6120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:11.393332 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "2c7e1f7d874148928890b621884db658"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "ba41fdc2ebfed6510266bba33c756b0d"
server_key_iv: "6679b06e4a21d1e3214c10e1aaf78889"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.395517 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "2c7e1f7d874148928890b621884db658"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "ba41fdc2ebfed6510266bba33c756b0d"
server_key_iv: "6679b06e4a21d1e3214c10e1aaf78889"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.405483 6120 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260619 02:51:11.441521 6135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.442018 6120 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260619 02:51:11.442166 6120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "2c7e1f7d874148928890b621884db658"
format_stamp: "Formatted at 2026-06-19 02:51:11 on dist-test-slave-0f9d"
server_key: "ba41fdc2ebfed6510266bba33c756b0d"
server_key_iv: "6679b06e4a21d1e3214c10e1aaf78889"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:11.442349 6120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:11.465420 6120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:11.470144 6120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:11.470467 6120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:11.470851 6120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:11.471421 6120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:11.471540 6120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.471597 6120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:11.471617 6120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:11.481716 6120 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:35553
I20260619 02:51:11.482419 6120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:51:11.483176 6248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:35553 every 8 connection(s)
I20260619 02:51:11.484630 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 6120
I20260619 02:51:11.484755 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:51:11.484947 27016 external_mini_cluster.cc:1428] Setting key 906bd7e8c1d4fc7b284c9189165f4127
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:51:11.497997 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.483817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:33849 (local address 127.26.98.62:36351)
0619 02:51:11.483945 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.483949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.484212 (+ 263us) server_negotiation.cc:407] Connection header received
0619 02:51:11.485110 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.485113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.485132 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.485166 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:11.486442 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.487352 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.489244 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.489575 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.489638 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.493729 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:11.493732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:11.493732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:11.493745 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:11.495745 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.496536 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.496538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.496539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.496622 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.497026 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.497027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.497027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.497201 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:11.497248 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.497745 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.497857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:51:11.498771 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.483588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36351 (local address 127.26.98.3:33849)
0619 02:51:11.484001 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.484076 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.484930 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.485640 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.485647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.485953 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:11.486324 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.486331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.487495 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.487497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.489099 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.489104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.489765 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.489766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.489958 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.491304 (+ 1346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:11.491322 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:11.493531 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:11.495934 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:11.495953 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:11.495958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:11.496421 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:11.496704 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:11.496705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:11.496705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:11.496940 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:11.497293 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:11.497300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:11.497561 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.498200 (+ 639us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.498455 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":111,"threads_started":1}
I20260619 02:51:11.499804 6249 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36351
I20260619 02:51:11.499985 6249 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:11.500267 6249 heartbeater.cc:511] Master 127.26.98.62:36351 requested a full tablet report, sending...
I20260619 02:51:11.500981 5765 ts_manager.cc:194] Registered new tserver with Master: 2c7e1f7d874148928890b621884db658 (127.26.98.3:35553)
I20260619 02:51:11.501569 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:33849
I20260619 02:51:11.508347 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:51:11.528995 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.511293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.26.98.62:36351)
0619 02:51:11.511356 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.511359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.519233 (+ 7874us) server_negotiation.cc:407] Connection header received
0619 02:51:11.519368 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.519370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.519386 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.519433 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:11.520313 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.521192 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.522693 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.523065 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.523131 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.525417 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:11.525420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:11.525421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:11.525440 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:11.527424 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.527949 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.527950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.527951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.528021 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:11.528218 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:11.528219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:11.528220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:11.528399 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:11.528489 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.528763 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.528867 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 02:51:11 dist-test-slave-0f9d krb5kdc[5288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837470, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:51:11.531988 5765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:51:11.534713 5765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:51:11.549626 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.541165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.26.98.1:33733)
0619 02:51:11.542050 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.542054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.544534 (+ 2480us) server_negotiation.cc:407] Connection header received
0619 02:51:11.544654 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.544658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.544677 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.544738 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:11.544984 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.546132 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.547628 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.548853 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.548948 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.549033 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.549446 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.549514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":66,"threads_started":1}
I20260619 02:51:11.550319 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.540561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33733 (local address 127.0.0.1:55578)
0619 02:51:11.544451 (+ 3890us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.544462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.544565 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.544860 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.544861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.544863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:11.544914 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.544915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.546239 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.546240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.547511 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.547515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.549168 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.549169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.549304 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.549321 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.550112 (+ 791us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.550225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3867,"thread_start_us":68,"threads_started":1}
I20260619 02:51:11.551013 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.540406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:35553 (local address 127.0.0.1:48630)
0619 02:51:11.542110 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.542120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.542168 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.542489 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.542490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.542492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:11.542551 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.542553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.544002 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.544003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.545941 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.545944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.550690 (+ 4746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.550692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.550769 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.550774 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.550850 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.550932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260619 02:51:11.551560 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0effce54014241a64f624afb38b22b (DEFAULT_TABLE table=test-table [id=e5aee9d9c1dc44c6a7e0214da3657adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:11.552083 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0effce54014241a64f624afb38b22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:11.553763 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.540974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.26.98.2:33649)
0619 02:51:11.541728 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.541735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.541746 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:51:11.541792 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.541795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.541813 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.541907 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:11.542158 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.543832 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.549475 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.550431 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.550519 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.550612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.553578 (+ 2966us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.553659 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":93,"threads_started":1}
I20260619 02:51:11.553891 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.540595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.26.98.3:35553)
0619 02:51:11.542304 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.542309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.542319 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:51:11.542370 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.542373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.542390 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.542613 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:11.542700 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.543896 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.546082 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.547373 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.551438 (+ 4065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.551598 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.551639 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.551708 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1664,"thread_start_us":100,"threads_started":1}
I20260619 02:51:11.554782 6164 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0effce54014241a64f624afb38b22b (DEFAULT_TABLE table=test-table [id=e5aee9d9c1dc44c6a7e0214da3657adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:11.555295 6164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0effce54014241a64f624afb38b22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:11.557411 6268 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Bootstrap starting.
I20260619 02:51:11.558300 6268 tablet_bootstrap.cc:654] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:11.558723 6268 log.cc:824] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:11.560199 6268 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: No bootstrap required, opened a new log
I20260619 02:51:11.560286 6268 ts_tablet_manager.cc:1397] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 02:51:11.563984 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.540147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:33649 (local address 127.0.0.1:42808)
0619 02:51:11.541405 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.541476 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.541563 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.541938 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.541943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.541946 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:11.542048 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.542051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.543940 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.543941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.549326 (+ 5385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.549329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.552386 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.552387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.552781 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.552792 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.552870 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.552943 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":96,"threads_started":1}
I20260619 02:51:11.563730 6268 raft_consensus.cc:348] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.564127 6268 raft_consensus.cc:374] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:11.564209 6268 raft_consensus.cc:729] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82035dbbf7144ab59943bf402b0e272d, State: Initialized, Role: FOLLOWER
I20260619 02:51:11.565469 6049 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0effce54014241a64f624afb38b22b (DEFAULT_TABLE table=test-table [id=e5aee9d9c1dc44c6a7e0214da3657adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:11.565995 6049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0effce54014241a64f624afb38b22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:11.564416 6268 consensus_queue.cc:260] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.571555 6268 ts_tablet_manager.cc:1428] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20260619 02:51:11.571794 5977 heartbeater.cc:503] Master 127.26.98.62:36351 was elected leader, sending a full tablet report...
I20260619 02:51:11.572871 6271 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Bootstrap starting.
I20260619 02:51:11.573208 6270 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Bootstrap starting.
I20260619 02:51:11.574688 6270 tablet_bootstrap.cc:654] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:11.575192 6270 log.cc:824] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:11.576097 6271 tablet_bootstrap.cc:654] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:11.576551 6271 log.cc:824] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:11.577584 6271 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: No bootstrap required, opened a new log
I20260619 02:51:11.577672 6271 ts_tablet_manager.cc:1397] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:51:11.578157 6270 tablet_bootstrap.cc:492] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: No bootstrap required, opened a new log
I20260619 02:51:11.578234 6270 ts_tablet_manager.cc:1397] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 02:51:11.580444 6270 raft_consensus.cc:348] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.580687 6270 raft_consensus.cc:374] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:11.580727 6270 raft_consensus.cc:729] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7e1f7d874148928890b621884db658, State: Initialized, Role: FOLLOWER
I20260619 02:51:11.580897 6270 consensus_queue.cc:260] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.581226 6270 ts_tablet_manager.cc:1428] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 02:51:11.581422 6249 heartbeater.cc:503] Master 127.26.98.62:36351 was elected leader, sending a full tablet report...
I20260619 02:51:11.580435 6271 raft_consensus.cc:348] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.582393 6271 raft_consensus.cc:374] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:11.582489 6271 raft_consensus.cc:729] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9243529888d84499a730b76f17a92a51, State: Initialized, Role: FOLLOWER
I20260619 02:51:11.582674 6271 consensus_queue.cc:260] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.583283 6271 ts_tablet_manager.cc:1428] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260619 02:51:11.583438 6115 heartbeater.cc:503] Master 127.26.98.62:36351 was elected leader, sending a full tablet report...
W20260619 02:51:11.601233 5978 tablet.cc:2369] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:51:11.668015 6116 tablet.cc:2369] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:51:11.713116 6275 raft_consensus.cc:482] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:51:11.713356 6275 raft_consensus.cc:504] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.713933 6275 leader_election.cc:288] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9243529888d84499a730b76f17a92a51 (127.26.98.2:33649), 82035dbbf7144ab59943bf402b0e272d (127.26.98.1:33733)
I20260619 02:51:11.718458 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.714500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:47243 (local address 127.26.98.1:33733)
0619 02:51:11.714551 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.714553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.714604 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:51:11.714729 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.714730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.714745 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.714778 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:11.715083 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.715835 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.717407 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.718012 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.718070 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.718085 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.718304 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.718373 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:51:11.718474 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.714196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:33649 (local address 127.26.98.3:47739)
0619 02:51:11.714253 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.714270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.714360 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.714756 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.714757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.714759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:11.714847 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.714849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.715945 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.715946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.717269 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.717272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.718144 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.718145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.718238 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.718242 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.718285 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.718352 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":8}
I20260619 02:51:11.718711 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.714306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33733 (local address 127.26.98.3:47243)
0619 02:51:11.714531 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.714541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:11.714635 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:11.714849 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:11.714850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:11.714852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:11.714982 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.714985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.715945 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.715946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.717262 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:11.717266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.718146 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.718147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:11.718227 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.718232 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:51:11.718579 (+ 347us) client_negotiation.cc:240] Negotiation successful
0619 02:51:11.718632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":37,"thread_start_us":145,"threads_started":1}
I20260619 02:51:11.719264 5931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0effce54014241a64f624afb38b22b" candidate_uuid: "2c7e1f7d874148928890b621884db658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82035dbbf7144ab59943bf402b0e272d" is_pre_election: true
I20260619 02:51:11.719393 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.714300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:47739 (local address 127.26.98.2:33649)
0619 02:51:11.714365 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.714367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.714380 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:51:11.714594 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.714596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.714613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.714673 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:11.714982 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.715817 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.717407 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.718026 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.718097 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.718110 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.719230 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.719301 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:51:11.719512 5931 raft_consensus.cc:2454] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c7e1f7d874148928890b621884db658 in term 0.
I20260619 02:51:11.719848 6136 leader_election.cc:302] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c7e1f7d874148928890b621884db658, 82035dbbf7144ab59943bf402b0e272d; no voters:
I20260619 02:51:11.719856 6069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0effce54014241a64f624afb38b22b" candidate_uuid: "2c7e1f7d874148928890b621884db658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9243529888d84499a730b76f17a92a51" is_pre_election: true
I20260619 02:51:11.720018 6069 raft_consensus.cc:2454] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c7e1f7d874148928890b621884db658 in term 0.
I20260619 02:51:11.720299 6275 raft_consensus.cc:2781] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:51:11.720374 6275 raft_consensus.cc:482] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:51:11.720396 6275 raft_consensus.cc:3037] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:11.721486 6275 raft_consensus.cc:504] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.721668 6275 leader_election.cc:288] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [CANDIDATE]: Term 1 election: Requested vote from peers 9243529888d84499a730b76f17a92a51 (127.26.98.2:33649), 82035dbbf7144ab59943bf402b0e272d (127.26.98.1:33733)
I20260619 02:51:11.721942 6069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0effce54014241a64f624afb38b22b" candidate_uuid: "2c7e1f7d874148928890b621884db658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9243529888d84499a730b76f17a92a51"
I20260619 02:51:11.721982 5931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0effce54014241a64f624afb38b22b" candidate_uuid: "2c7e1f7d874148928890b621884db658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82035dbbf7144ab59943bf402b0e272d"
I20260619 02:51:11.722064 6069 raft_consensus.cc:3037] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:11.722081 5931 raft_consensus.cc:3037] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:11.723300 5931 raft_consensus.cc:2454] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c7e1f7d874148928890b621884db658 in term 1.
I20260619 02:51:11.723491 6069 raft_consensus.cc:2454] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c7e1f7d874148928890b621884db658 in term 1.
I20260619 02:51:11.723668 6136 leader_election.cc:302] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c7e1f7d874148928890b621884db658, 82035dbbf7144ab59943bf402b0e272d; no voters:
I20260619 02:51:11.723807 6275 raft_consensus.cc:2781] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:51:11.724041 6275 raft_consensus.cc:686] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 1 LEADER]: Becoming Leader. State: Replica: 2c7e1f7d874148928890b621884db658, State: Running, Role: LEADER
I20260619 02:51:11.724175 6275 consensus_queue.cc:237] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } }
I20260619 02:51:11.725412 5764 catalog_manager.cc:5495] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c7e1f7d874148928890b621884db658 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "2c7e1f7d874148928890b621884db658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c7e1f7d874148928890b621884db658" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 35553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 } health_report { overall_health: UNKNOWN } } }
I20260619 02:51:11.733618 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.729616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.26.98.3:35553)
0619 02:51:11.729671 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:11.729674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:11.729685 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:51:11.729747 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:11.729748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:11.729789 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:11.729905 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:51:11.730251 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.731080 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.732266 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:11.732607 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:11.732670 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:11.732908 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:51:11.733087 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:51:11.733172 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:11.733484 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 02:51:11.733513 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 02:51:11.733844 6250 tablet.cc:2369] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:51:11.736828 5931 raft_consensus.cc:1261] T 0c0effce54014241a64f624afb38b22b P 82035dbbf7144ab59943bf402b0e272d [term 1 FOLLOWER]: Refusing update from remote peer 2c7e1f7d874148928890b621884db658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:51:11.736881 6069 raft_consensus.cc:1261] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 1 FOLLOWER]: Refusing update from remote peer 2c7e1f7d874148928890b621884db658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:51:11.737282 6275 consensus_queue.cc:1035] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9243529888d84499a730b76f17a92a51" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:51:11.737462 6278 consensus_queue.cc:1035] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82035dbbf7144ab59943bf402b0e272d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:51:11.741156 6281 mvcc.cc:205] Tried to move back new op lower bound from 7298406284230180864 to 7298406284183240704. Current Snapshot: MvccSnapshot[applied={T|T < 7298406284230180864}]
I20260619 02:51:11.742401 6282 mvcc.cc:205] Tried to move back new op lower bound from 7298406284230180864 to 7298406284183240704. Current Snapshot: MvccSnapshot[applied={T|T < 7298406284230180864}]
I20260619 02:51:11.744098 6283 mvcc.cc:205] Tried to move back new op lower bound from 7298406284230180864 to 7298406284183240704. Current Snapshot: MvccSnapshot[applied={T|T < 7298406284230180864}]
I20260619 02:51:11.746340 5763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:51:11.746484 5763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:51:11.747547 5763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d03ea4d94d945cbb926e55d0544948b: Sending DeleteTablet for 3 replicas of tablet 0c0effce54014241a64f624afb38b22b
I20260619 02:51:11.748216 6049 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0effce54014241a64f624afb38b22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260619 02:51:11.748456 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 5846
I20260619 02:51:11.748594 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0effce54014241a64f624afb38b22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:55578
I20260619 02:51:11.749485 6164 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0effce54014241a64f624afb38b22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:11 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:48630
W20260619 02:51:11.755005 5749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:33733 (error 108)
I20260619 02:51:11.755550 6291 tablet_replica.cc:330] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: stopping tablet replica
I20260619 02:51:11.755776 6291 raft_consensus.cc:2229] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:51:11.756088 6291 raft_consensus.cc:2258] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:51:11.756280 6290 tablet_replica.cc:330] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51: stopping tablet replica
I20260619 02:51:11.756544 6290 raft_consensus.cc:2229] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:51:11.756897 6290 raft_consensus.cc:2258] T 0c0effce54014241a64f624afb38b22b P 9243529888d84499a730b76f17a92a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:51:11.757326 6291 ts_tablet_manager.cc:1905] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:51:11.757685 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 5985
I20260619 02:51:11.760511 6291 ts_tablet_manager.cc:1918] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:51:11.760641 6291 log.cc:1196] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/0c0effce54014241a64f624afb38b22b
I20260619 02:51:11.760977 6291 ts_tablet_manager.cc:1939] T 0c0effce54014241a64f624afb38b22b P 2c7e1f7d874148928890b621884db658: Deleting consensus metadata
I20260619 02:51:11.762223 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.761986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33733 (local address 127.0.0.1:55594)
0619 02:51:11.762065 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.762145 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:51:11.762769 5749 catalog_manager.cc:4584] TS 82035dbbf7144ab59943bf402b0e272d (127.26.98.1:33733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c0effce54014241a64f624afb38b22b: Network error: Client connection negotiation failed: client connection to 127.26.98.1:33733: connect: Connection refused (error 111)
I20260619 02:51:11.765179 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:11.765032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:33649 (local address 127.0.0.1:42822)
0619 02:51:11.765099 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:11.765129 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:33649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:51:11.765741 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 6120
I20260619 02:51:11.765717 5749 catalog_manager.cc:4841] TS 2c7e1f7d874148928890b621884db658 (127.26.98.3:35553): tablet 0c0effce54014241a64f624afb38b22b (table test-table [id=e5aee9d9c1dc44c6a7e0214da3657adf]) successfully deleted
I20260619 02:51:11.772169 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 5733
I20260619 02:51:11.777575 27016 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:51:12.123957 27016 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:51:12.124111 27016 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:51:12.470984 27016 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:51:12.471 UTC [5318] LOG: received smart shutdown request
2026-06-19 02:51:12.475 UTC [5318] LOG: background worker "logical replication launcher" (PID 5327) exited with exit code 1
2026-06-19 02:51:12.476 UTC [5322] LOG: shutting down
2026-06-19 02:51:12.691 UTC [5318] LOG: database system is shut down
2026-06-19T02:51:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (60952 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:51:12 dist-test-slave-0f9d krb5kdc[6301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:51:12 dist-test-slave-0f9d krb5kdc[6301](info): set up 2 sockets
Jun 19 02:51:12 dist-test-slave-0f9d krb5kdc[6301](info): commencing operation
krb5kdc: starting...
W20260619 02:51:13.811299 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:51:13 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:51:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangeradmin_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangeradmin_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangerlookup_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangerlookup_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
I20260619 02:51:14.015614 27016 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcBfLLw/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/postgres -l logfile start
2026-06-19 02:51:17.182 UTC [6331] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:51:17.182 UTC [6331] LOG: listening on IPv4 address "127.26.98.20", port 42545
2026-06-19 02:51:17.189 UTC [6331] LOG: listening on Unix socket "/tmp/.s.PGSQL.42545"
2026-06-19 02:51:17.214 UTC [6334] LOG: database system was shut down at 2026-06-19 02:51:15 UTC
2026-06-19 02:51:17.220 UTC [6331] LOG: database system is ready to accept connections
I20260619 02:51:18.127519 27016 mini_postgres.cc:96] Postgres bound to 42545
2026-06-19 02:51:18.135 UTC [6342] FATAL: database "slave" does not exist
127.26.98.20:42545 - accepting connections
I20260619 02:51:18.136607 27016 mini_ranger.cc:161] Starting Ranger...
I20260619 02:51:18.161078 27016 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:51:18.785827 27016 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:51:18.785933 27016 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin
2026-06-19 02:51:19,090 [I] DB FLAVOR :POSTGRES
2026-06-19 02:51:19,091 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:51:19,091 [I] Checking connection..
2026-06-19 02:51:19,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:51:19,439 [I] Checking connection passed.
2026-06-19 02:51:19,439 [I] --------- Verifying version history table ---------
2026-06-19 02:51:19,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:19,768 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:51:19,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:20,062 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:51:20,063 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:51:20,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:51:20.382 UTC [6406] WARNING: there is no transaction in progress
2026-06-19 02:51:20,390 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:51:20,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:20,687 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:51:20,688 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:20,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:51:20,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:51:21,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0f9d.c.gcp-upstream.internal','N') ;"
2026-06-19 02:51:21,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:51:21,976 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:51:21,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:51:22,302 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:51:22,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:51:22,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:51:22,917 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:51:22,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:51:24.199 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.226 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.253 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.279 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.302 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.329 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.423 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.434 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.453 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.467 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.481 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.501 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.516 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.527 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.540 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:24.551 UTC [6514] WARNING: there is no transaction in progress
2026-06-19 02:51:25,565 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:51:25,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:51:25,859 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:51:25,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:51:26,180 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:51:26,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:51:26,511 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:51:26,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:51:26,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:51:27,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0f9d.c.gcp-upstream.internal';"
2026-06-19 02:51:27,445 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:51:27,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:51:27,774 [I] DB_PATCHES have already been applied
I20260619 02:51:27.782932 27016 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcBfLLw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:51:27.783030 27016 mini_ranger.cc:193] Using host: 127.26.98.20
I20260619 02:51:27.786586 27016 mini_ranger.cc:227] Ranger admin URL: http://127.26.98.20:40145
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:51:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:51:29.969828 27016 mini_ranger.cc:160] Time spent starting Ranger: real 11.833s user 0.000s sys 0.014s
Jun 19, 2026 2:51:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:51:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:51:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:51:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:51:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:51:56.575111 27016 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.26.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.26.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangerkms_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.26.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/rangerkms_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.26.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/HTTP_127.26.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:51:56 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837516, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:51:56.686179 27016 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:51:56.700316 27016 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:51:57.725054 27016 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:51:57.725148 27016 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms
I20260619 02:51:57.725167 27016 mini_ranger_kms.cc:227] Using postgres at 127.26.98.20:42545
2026-06-19 02:51:57,937 [I] DB FLAVOR :POSTGRES
2026-06-19 02:51:57,937 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:51:57,937 [I] Checking connection
2026-06-19 02:51:57,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:51:58,296 [I] connection success
2026-06-19 02:51:58,296 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:51:58,296 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:51:58,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:51:58,662 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:51:58,663 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:58,663 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:51:58,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.98.20:42545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:51:59,023 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:51:59.678454 27016 mini_ranger_kms.cc:310] Created kms service
I20260619 02:51:59.947407 27016 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:52:00.007185 27016 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:52:00 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837516, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:00.592123 27016 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:52:00.592468 27016 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcBfLLw/build/release/bin/postgresql.jar:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcBfLLw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:52:00.592495 27016 mini_ranger_kms.cc:240] Using host: 127.26.98.20
I20260619 02:52:00.597002 27016 mini_ranger_kms.cc:276] Ranger KMS PID: 6696
I20260619 02:52:00.597126 27016 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.26.98.20:40631
Jun 19, 2026 2:52:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:52:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:52:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:52:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:52:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:52:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:52:02.781061 27016 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.095s user 0.016s sys 0.178s
I20260619 02:52:02.781183 27016 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:52:02.781237 27016 mini_ranger_kms.cc:401] 127.26.98.20:40631/kms/v1/keys
Jun 19 02:52:03 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:52:03 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
Jun 19 02:52:04 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.98.20@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
Jun 19 02:52:07 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:52:07.635926 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36429
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36429
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:07.736366 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:07.736701 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:07.736738 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:07.738646 6746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:07.738767 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:07.738791 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:07.738808 6746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:07.738823 6746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:07.742028 6746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
--ranger_kms_url=127.26.98.20:40631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36429
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:07.742508 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:07.742816 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:07.750777 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:07.751060 6746 server_base.cc:1034] running on GCE node
W20260619 02:52:07.750954 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:07.750777 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:07.751830 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:07.752283 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:07.753484 6746 hybrid_clock.cc:648] HybridClock initialized: now 1781837527753461 us; error 52 us; skew 500 ppm
Jun 19 02:52:07 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:07.757282 6746 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:07.757987 6746 webserver.cc:460] Webserver started at http://127.26.98.62:35031/ using document root <none> and password file <none>
I20260619 02:52:07.758231 6746 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:07.758278 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:07.758430 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:52:07 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:07.833562 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "562414d2563e4593a39a368a63493cfc"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-0f9d"
server_key: "e9274d3ba0c7b927ff6b0fec5de05aeb"
server_key_iv: "72d52aa46b97a31c0937bd8e3f9d55aa"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:07.834107 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "562414d2563e4593a39a368a63493cfc"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-0f9d"
server_key: "e9274d3ba0c7b927ff6b0fec5de05aeb"
server_key_iv: "72d52aa46b97a31c0937bd8e3f9d55aa"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:07.835819 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:52:07.853011 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:07.853266 6746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:07.853389 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "562414d2563e4593a39a368a63493cfc"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-0f9d"
server_key: "e9274d3ba0c7b927ff6b0fec5de05aeb"
server_key_iv: "72d52aa46b97a31c0937bd8e3f9d55aa"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:07.853505 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:07.921926 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:07.924484 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:07.924685 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:07.930457 6746 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36429
I20260619 02:52:07.930488 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36429 every 8 connection(s)
I20260619 02:52:07.931164 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:07.932121 6814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:07.935849 6814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Bootstrap starting.
I20260619 02:52:07.936744 6814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:07.937018 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 6746
I20260619 02:52:07.937139 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:52:07.937325 6814 log.cc:824] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:07.937363 27016 external_mini_cluster.cc:1428] Setting key c30d67118aed930dd54125c677ca70c1
I20260619 02:52:07.938494 6814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: No bootstrap required, opened a new log
I20260619 02:52:07.942585 6814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:07.942979 6814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:07.943024 6814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562414d2563e4593a39a368a63493cfc, State: Initialized, Role: FOLLOWER
I20260619 02:52:07.943251 6814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:07.943485 6814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:07.943697 6814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:07.943775 6814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:07.944887 6814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:07.945107 6814 leader_election.cc:302] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 562414d2563e4593a39a368a63493cfc; no voters:
I20260619 02:52:07.945338 6814 leader_election.cc:288] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:07.945451 6819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:07.945976 6819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 LEADER]: Becoming Leader. State: Replica: 562414d2563e4593a39a368a63493cfc, State: Running, Role: LEADER
I20260619 02:52:07.946115 6814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:07.946194 6819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:07.946981 6819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "562414d2563e4593a39a368a63493cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } } }
I20260619 02:52:07.947229 6819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:07.947432 6819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 562414d2563e4593a39a368a63493cfc. Latest consensus state: current_term: 1 leader_uuid: "562414d2563e4593a39a368a63493cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } } }
I20260619 02:52:07.947546 6819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:07.947716 6828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:07.948253 6814 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 19 02:52:07 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:07.954471 6828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:07.958806 6828 catalog_manager.cc:1347] Generated new cluster ID: ea06aa5670e04d77805ca6bc56d65e63
I20260619 02:52:07.958910 6828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:07.965811 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:07.938066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.26.98.62:36429)
0619 02:52:07.939022 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:07.939035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:07.939084 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:52:07.940166 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:07.940181 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:07.940274 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:07.940632 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:07.941711 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:07.943721 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:07.945019 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:07.945938 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:07.948373 (+ 2435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:07.953545 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:07.953739 (+ 194us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:07.953740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:07.953860 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:07.960077 (+ 6217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:07.960722 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:07.960727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:07.960727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:07.960856 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:07.963271 (+ 2415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:07.963271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:07.963272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:07.963742 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:07.964652 (+ 910us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:07.964889 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:52:07.965120 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":116,"threads_started":1}
I20260619 02:52:08.010998 6828 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:08.012049 6828 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:08.022053 6828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Generated new TSK 0
I20260619 02:52:08.022516 6828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:52:08.922765 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:09.036518 6859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.036896 6859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.036945 6859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.038700 6859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.038785 6859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.038836 6859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:09.041074 6859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:09.041441 6859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.041725 6859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.047756 6865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.047911 6864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.047864 6867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.048629 6859 server_base.cc:1034] running on GCE node
I20260619 02:52:09.048918 6859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.049264 6859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.050484 6859 hybrid_clock.cc:648] HybridClock initialized: now 1781837529050439 us; error 65 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.054361 6859 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:09.055038 6859 webserver.cc:460] Webserver started at http://127.26.98.1:43171/ using document root <none> and password file <none>
I20260619 02:52:09.055444 6859 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.055533 6859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.055693 6859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:09.076622 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "af93142dc84e4a60ba572a548edd7dc3"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "94c68c97e04d7e1204927cded9afa3f7"
server_key_iv: "151903f5e363254114cded3c04558439"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.077301 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "af93142dc84e4a60ba572a548edd7dc3"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "94c68c97e04d7e1204927cded9afa3f7"
server_key_iv: "151903f5e363254114cded3c04558439"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.079722 6859 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:09.097327 6874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.097653 6859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:09.097769 6859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "af93142dc84e4a60ba572a548edd7dc3"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "94c68c97e04d7e1204927cded9afa3f7"
server_key_iv: "151903f5e363254114cded3c04558439"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.097913 6859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.129027 6859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.133330 6859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.133786 6859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.134215 6859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.135196 6859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.135311 6859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.135432 6859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.135510 6859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.147073 6859 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36575
I20260619 02:52:09.147225 6989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36575 every 8 connection(s)
I20260619 02:52:09.147835 6859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:09.151928 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 6859
I20260619 02:52:09.152067 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:52:09.152284 27016 external_mini_cluster.cc:1428] Setting key beeca6bdca6754382eb856f4f38589dd
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:09.166641 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.149375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:36905 (local address 127.26.98.62:36429)
0619 02:52:09.151016 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.151018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.151029 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:09.151068 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.151069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.151089 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.152550 (+ 1461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:09.152620 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.153608 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.157342 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.157705 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.157817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.162263 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:09.162266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:09.162267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:09.162279 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.164465 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.165234 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.165237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.165237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.165329 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.165658 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.165659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.165660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.165850 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.165910 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.166381 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.166504 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":82,"threads_started":1}
I20260619 02:52:09.167841 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.148978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.1:36905)
0619 02:52:09.149251 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.149318 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.150634 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.151335 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.151343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.151878 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:09.152326 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.152340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.153790 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.153791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.156531 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.156543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.158285 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.158291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.158711 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.159755 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:09.159770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:09.162050 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:09.164646 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.164670 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.164675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.165130 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.165406 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.165406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.165407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.165577 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.165960 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:09.165969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:09.166217 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.167282 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.167532 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260619 02:52:09.168877 6990 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:09.169019 6990 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:09.169296 6990 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:52:09.170310 6778 ts_manager.cc:194] Registered new tserver with Master: af93142dc84e4a60ba572a548edd7dc3 (127.26.98.1:36575)
I20260619 02:52:09.171314 6778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:36905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:52:09.207880 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:52:09.360095 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.360601 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.360687 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.363384 6996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.363591 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.363675 6996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:09.367050 6996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:09.367638 6996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.368021 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.376735 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.376988 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.377482 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.380167 6996 server_base.cc:1034] running on GCE node
I20260619 02:52:09.380682 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.381021 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.382193 6996 hybrid_clock.cc:648] HybridClock initialized: now 1781837529382173 us; error 41 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.386920 6996 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:09.387696 6996 webserver.cc:460] Webserver started at http://127.26.98.2:43571/ using document root <none> and password file <none>
I20260619 02:52:09.388027 6996 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.388093 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.388266 6996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:09.409747 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "05f1f38602d14b36ae94e806cfc34443"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "c33dbadaa409c2f5d5d596ece358f31e"
server_key_iv: "c57d94f08de2f605346bd73d26b83c6f"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.410662 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "05f1f38602d14b36ae94e806cfc34443"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "c33dbadaa409c2f5d5d596ece358f31e"
server_key_iv: "c57d94f08de2f605346bd73d26b83c6f"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.413996 6996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:09.437155 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.437558 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:52:09.437711 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "05f1f38602d14b36ae94e806cfc34443"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "c33dbadaa409c2f5d5d596ece358f31e"
server_key_iv: "c57d94f08de2f605346bd73d26b83c6f"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.437963 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.456691 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.460973 6996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.461527 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.462178 6996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.463112 6996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.463248 6996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.463400 6996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.463490 6996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.476663 6996 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:34149
I20260619 02:52:09.477499 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:09.478264 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 6996
I20260619 02:52:09.478375 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:52:09.478974 27016 external_mini_cluster.cc:1428] Setting key e91790f08e23e8dfffffbcc6c972d934
I20260619 02:52:09.482981 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:34149 every 8 connection(s)
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:09.511716 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.486589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:38257 (local address 127.26.98.62:36429)
0619 02:52:09.486649 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.486652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.487065 (+ 413us) server_negotiation.cc:407] Connection header received
0619 02:52:09.488061 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.488063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.488081 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.488113 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:09.489278 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.490145 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.492039 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.492409 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.492471 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.503263 (+ 10792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:09.503266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:09.503266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:09.503278 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.509034 (+ 5756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.509906 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.509908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.509909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.510001 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.510370 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.510371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.510372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.510555 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.510620 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.511473 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.511594 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:52:09.512633 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.486018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.2:38257)
0619 02:52:09.486829 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.486933 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.487905 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.488377 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.488383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.488695 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:09.489060 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.489069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.490311 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.490313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.491886 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.491891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.492657 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.492658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.493005 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.493897 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:09.493911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:09.503044 (+ 9133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:09.509209 (+ 6165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.509231 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.509236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.509767 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.510079 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.510081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.510081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.510285 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.510654 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:09.510662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:09.510972 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.511101 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.512218 (+ 1117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":262,"threads_started":1}
I20260619 02:52:09.513540 7128 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:09.513676 7128 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:09.513936 7128 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:09.514456 6778 ts_manager.cc:194] Registered new tserver with Master: 05f1f38602d14b36ae94e806cfc34443 (127.26.98.2:34149)
I20260619 02:52:09.515036 6778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:38257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:52:09.535374 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:09.701184 7133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.701790 7133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.701964 7133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.704610 7133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.704844 7133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.704946 7133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:09.707846 7133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:09.708577 7133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.709050 7133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.717870 7141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.718231 7133 server_base.cc:1034] running on GCE node
W20260619 02:52:09.719013 7139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.719329 7138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.719712 7133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.720091 7133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.721305 7133 hybrid_clock.cc:648] HybridClock initialized: now 1781837529721283 us; error 42 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.727711 7133 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:09.728541 7133 webserver.cc:460] Webserver started at http://127.26.98.3:39103/ using document root <none> and password file <none>
I20260619 02:52:09.733603 7133 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.733804 7133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.734207 7133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:09.754541 7133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "ed860430fd674137a7eab569e1fc5be2"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "d87229cb25b1daab9bd694d064ea0f1a"
server_key_iv: "1ae1d9cc4ee1c4777061678b6a6a5a1e"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.755573 7133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "ed860430fd674137a7eab569e1fc5be2"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "d87229cb25b1daab9bd694d064ea0f1a"
server_key_iv: "1ae1d9cc4ee1c4777061678b6a6a5a1e"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.758181 7133 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:09.783262 7148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.783601 7133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:52:09.783780 7133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "ed860430fd674137a7eab569e1fc5be2"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "d87229cb25b1daab9bd694d064ea0f1a"
server_key_iv: "1ae1d9cc4ee1c4777061678b6a6a5a1e"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:09.783955 7133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.836694 7133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.840837 7133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.841061 7133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.841445 7133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.841941 7133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.841998 7133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.842039 7133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.842061 7133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.850078 7133 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:36905
I20260619 02:52:09.850728 7133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:09.855696 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7133
I20260619 02:52:09.855840 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:52:09.856078 27016 external_mini_cluster.cc:1428] Setting key f25803e10f9bf081b1fcbefa4ec02530
I20260619 02:52:09.861361 7261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:36905 every 8 connection(s)
Jun 19 02:52:09 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:09.875675 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.862378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:35609 (local address 127.26.98.62:36429)
0619 02:52:09.862764 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.862766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.862869 (+ 103us) server_negotiation.cc:407] Connection header received
0619 02:52:09.863757 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.863759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.863774 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.863808 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:09.864843 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.865708 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.867617 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.867917 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.867971 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.871492 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:09.871494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:09.871495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:09.871507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.873505 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.874241 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.874242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.874243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.874320 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.874646 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.874647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.874647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.874821 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.874923 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.875414 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.875531 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260619 02:52:09.876771 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.862434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.3:35609)
0619 02:52:09.862689 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.862751 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.863608 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.864059 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.864065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.864368 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:09.864714 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.864722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.865840 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.865841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.867483 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.867488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.868115 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.868116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.868308 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.869107 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:09.869122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:09.871320 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:09.873666 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.873684 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.873689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.874132 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.874393 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.874394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.874395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.874570 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.874967 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:09.874976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:09.875233 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.876199 (+ 966us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.876456 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":132,"threads_started":1}
I20260619 02:52:09.877882 7262 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:09.878024 7262 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:09.878293 7262 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:09.878988 6778 ts_manager.cc:194] Registered new tserver with Master: ed860430fd674137a7eab569e1fc5be2 (127.26.98.3:36905)
I20260619 02:52:09.879529 6778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:35609
I20260619 02:52:09.881440 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:09.882570 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 6859
I20260619 02:52:09.892575 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 6996
I20260619 02:52:09.899825 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7133
I20260619 02:52:09.908025 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 6746
I20260619 02:52:09.914167 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36429
--webserver_interface=127.26.98.62
--webserver_port=35031
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36429
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:10.011452 7265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:10.011850 7265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:10.011901 7265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:10.013624 7265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:10.013715 7265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:10.013729 7265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:10.013741 7265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:10.013752 7265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:10.016207 7265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36429
--ranger_config_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:10.016608 7265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:10.016939 7265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:10.024410 7271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.024668 7265 server_base.cc:1034] running on GCE node
W20260619 02:52:10.024390 7273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.024390 7270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.025233 7265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:10.025704 7265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:10.027015 7265 hybrid_clock.cc:648] HybridClock initialized: now 1781837530026988 us; error 45 us; skew 500 ppm
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:10.030592 7265 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:10.031286 7265 webserver.cc:460] Webserver started at http://127.26.98.62:35031/ using document root <none> and password file <none>
I20260619 02:52:10.031570 7265 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:10.031656 7265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:10.083854 7265 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:52:10.085111 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.085395 7265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:10.085532 7265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "562414d2563e4593a39a368a63493cfc"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-0f9d"
server_key: "e9274d3ba0c7b927ff6b0fec5de05aeb"
server_key_iv: "72d52aa46b97a31c0937bd8e3f9d55aa"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:10.086367 7265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:10.103681 7265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:10.106807 7265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:10.107091 7265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:10.113780 7265 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36429
I20260619 02:52:10.113843 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36429 every 8 connection(s)
I20260619 02:52:10.114491 7265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:10.116250 7334 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:10.117273 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Bootstrap starting.
I20260619 02:52:10.122566 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7265
I20260619 02:52:10.123123 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:36575
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:36429
--webserver_port=43171
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:10.131692 7334 log.cc:824] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:10.134358 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:10.134713 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Bootstrap complete.
I20260619 02:52:10.137466 7334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:10.137813 7334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562414d2563e4593a39a368a63493cfc, State: Initialized, Role: FOLLOWER
I20260619 02:52:10.138031 7334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:10.138119 7334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:10.138196 7334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:10.138255 7334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:52:10.139880 7334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:10.140038 7334 leader_election.cc:302] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 562414d2563e4593a39a368a63493cfc; no voters:
I20260619 02:52:10.140265 7334 leader_election.cc:288] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:52:10.140326 7338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:52:10.140460 7338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [term 2 LEADER]: Becoming Leader. State: Replica: 562414d2563e4593a39a368a63493cfc, State: Running, Role: LEADER
I20260619 02:52:10.140571 7338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } }
I20260619 02:52:10.140618 7334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:10.140970 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 562414d2563e4593a39a368a63493cfc. Latest consensus state: current_term: 2 leader_uuid: "562414d2563e4593a39a368a63493cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } } }
I20260619 02:52:10.141353 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "562414d2563e4593a39a368a63493cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562414d2563e4593a39a368a63493cfc" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36429 } } }
I20260619 02:52:10.141500 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:10.141455 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:10.141767 7347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:10.142534 7347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:10.142722 7347 catalog_manager.cc:1259] Loaded cluster ID: ea06aa5670e04d77805ca6bc56d65e63
I20260619 02:52:10.142765 7347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:10.143112 7334 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:52:10.146643 7347 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:10.147715 7347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Loaded TSK: 0
I20260619 02:52:10.148298 7347 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:10.237457 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:10.237929 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:10.237977 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:10.241451 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:10.241573 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:10.241636 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:10.245177 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:36575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=43171
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:10.245633 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:10.245939 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:10.253419 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.253453 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.253937 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.255256 7336 server_base.cc:1034] running on GCE node
I20260619 02:52:10.255692 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:10.256085 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:10.259443 7336 hybrid_clock.cc:648] HybridClock initialized: now 1781837530259416 us; error 56 us; skew 500 ppm
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:10.263563 7336 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:10.264295 7336 webserver.cc:460] Webserver started at http://127.26.98.1:43171/ using document root <none> and password file <none>
I20260619 02:52:10.264628 7336 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:10.264712 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:10.291749 7336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:52:10.292642 7382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.292815 7336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:10.292924 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "af93142dc84e4a60ba572a548edd7dc3"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "94c68c97e04d7e1204927cded9afa3f7"
server_key_iv: "151903f5e363254114cded3c04558439"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:10.293444 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:10.313985 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:10.319522 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:10.320042 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:10.320444 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:10.320973 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:10.321027 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.321067 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:10.321087 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.329780 7336 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36575
I20260619 02:52:10.330298 7495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36575 every 8 connection(s)
I20260619 02:52:10.330440 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:10.339113 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7336
I20260619 02:52:10.339473 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:34149
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:36429
--webserver_port=43571
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:10.348810 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.331426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.1:35697)
0619 02:52:10.332328 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.332411 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:10.333147 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:10.334759 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:10.334767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:10.335334 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:10.335690 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.335701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.337689 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.337692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.339358 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.339364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.340624 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.340625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.340803 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.341516 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:10.341529 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:10.343537 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:10.346214 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.346231 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.346235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.346647 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.347106 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.347107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.347108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.347294 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.347944 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:10.347952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:10.348162 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 02:52:10.348273 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:52:10.348502 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":125,"threads_started":1}
I20260619 02:52:10.350179 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.331921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:35697 (local address 127.26.98.62:36429)
0619 02:52:10.333434 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:10.333445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:10.333491 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:52:10.333662 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:10.333668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:10.333736 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:10.334415 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:10.335878 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.337472 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.339521 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.340283 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.340452 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.343733 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:10.343740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:10.343740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:10.343757 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:10.346049 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.346780 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.346786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.346787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.347014 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.347377 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.347379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.347379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.347795 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:10.347908 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:10.349683 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0619 02:52:10.349882 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":99,"threads_started":1}
I20260619 02:52:10.352082 7496 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:10.352259 7496 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.352577 7496 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.354050 7496 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36429 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:10.354187 7496 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.354291 7496 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:10.354801 7496 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.354933 7496 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.355387 7496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:52:10.503253 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:10.503701 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:10.503757 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:10.506187 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:10.506295 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:10.506356 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:10.509661 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:34149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=43571
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:10.510248 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:10.510811 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:10.517472 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.517472 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.519095 7500 server_base.cc:1034] running on GCE node
W20260619 02:52:10.519619 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.520110 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:10.520757 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:10.521986 7500 hybrid_clock.cc:648] HybridClock initialized: now 1781837530521970 us; error 43 us; skew 500 ppm
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:10.526484 7500 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:10.527452 7500 webserver.cc:460] Webserver started at http://127.26.98.2:43571/ using document root <none> and password file <none>
I20260619 02:52:10.527771 7500 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:10.527855 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:10.553974 7500 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260619 02:52:10.555895 7516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.556210 7500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:10.556344 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "05f1f38602d14b36ae94e806cfc34443"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "c33dbadaa409c2f5d5d596ece358f31e"
server_key_iv: "c57d94f08de2f605346bd73d26b83c6f"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:10.556994 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:10.578333 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:10.581564 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:10.581781 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:10.582144 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:10.582612 7500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:10.582660 7500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.582716 7500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:10.582736 7500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.593333 7500 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:34149
I20260619 02:52:10.594194 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:10.595088 7629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:34149 every 8 connection(s)
I20260619 02:52:10.602322 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7500
I20260619 02:52:10.602698 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.98.20:40631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:36905
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:36429
--webserver_port=39103
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:10.628389 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.595302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.2:47173)
0619 02:52:10.604910 (+ 9608us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.605023 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:10.605985 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:10.606510 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:10.606516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:10.606959 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:10.607354 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.607364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.608507 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.608508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.610032 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.610037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.610738 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.610739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.610975 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.613657 (+ 2682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:10.613675 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:10.623494 (+ 9819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:10.626085 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.626107 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.626112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.626577 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.626835 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.626836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.626836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.627061 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.627433 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:10.627441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:10.627709 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 02:52:10.627834 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:52:10.628079 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9515,"thread_start_us":99,"threads_started":1}
I20260619 02:52:10.629022 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.595712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:47173 (local address 127.26.98.62:36429)
0619 02:52:10.595775 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:10.595778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:10.605217 (+ 9439us) server_negotiation.cc:407] Connection header received
0619 02:52:10.606183 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:10.606186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:10.606204 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:10.606241 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:10.607506 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.608334 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.610194 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.610543 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.610596 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.623771 (+ 13175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:10.623775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:10.623775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:10.623789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:10.625903 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.626685 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.626687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.626688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.626767 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.627146 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.627147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.627147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.627327 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:10.627399 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:10.628785 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0619 02:52:10.628924 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:52:10.629516 7630 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:10.629642 7630 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.629890 7630 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.630673 7630 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36429 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:10.630759 7630 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.630832 7630 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:10.631292 7630 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.631382 7630 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.631670 7630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:52:10.762061 7633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:10.762584 7633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:10.762684 7633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:10.765143 7633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:10.765239 7633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:10.765292 7633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:10.768548 7633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.98.20:40631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:36905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=39103
--tserver_master_addrs=127.26.98.62:36429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:10.769028 7633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:10.769381 7633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:10.776928 7639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.777361 7638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:10.777958 7641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:10.778535 7633 server_base.cc:1034] running on GCE node
I20260619 02:52:10.778825 7633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:10.779258 7633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:10.780457 7633 hybrid_clock.cc:648] HybridClock initialized: now 1781837530780431 us; error 49 us; skew 500 ppm
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:10.784891 7633 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:10.785785 7633 webserver.cc:460] Webserver started at http://127.26.98.3:39103/ using document root <none> and password file <none>
I20260619 02:52:10.786130 7633 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:10.786228 7633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:10.816286 7633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:52:10.820962 7650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.821318 7633 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260619 02:52:10.821450 7633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "ed860430fd674137a7eab569e1fc5be2"
format_stamp: "Formatted at 2026-06-19 02:52:09 on dist-test-slave-0f9d"
server_key: "d87229cb25b1daab9bd694d064ea0f1a"
server_key_iv: "1ae1d9cc4ee1c4777061678b6a6a5a1e"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:10.822149 7633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:10.846427 7633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:10.849038 7633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:10.849196 7633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:10.849490 7633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:10.849954 7633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:10.849995 7633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.850023 7633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:10.850047 7633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:10.860684 7633 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:36905
I20260619 02:52:10.860740 7763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:36905 every 8 connection(s)
I20260619 02:52:10.861208 7633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:10.868144 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7633
Jun 19 02:52:10 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837530, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:10.877724 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.862519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:35223 (local address 127.26.98.62:36429)
0619 02:52:10.862593 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:10.862596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:10.863010 (+ 414us) server_negotiation.cc:407] Connection header received
0619 02:52:10.863994 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:10.863996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:10.864015 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:10.864051 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:10.865186 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.866017 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.867896 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.868229 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.868292 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.872942 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:10.872944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:10.872944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:10.872957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:10.875182 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.875937 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.875939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.875940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.876006 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.876376 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.876377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.876378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.876551 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:10.876678 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:10.877458 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 02:52:10.877589 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:52:10.878808 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.862536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36429 (local address 127.26.98.3:35223)
0619 02:52:10.862774 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.862830 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:10.863780 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:10.864316 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:10.864322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:10.864656 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:10.865040 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.865049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.866183 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.866184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.867728 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:10.867734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.868440 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.868442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:10.868932 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.869704 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:10.869717 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:10.872026 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:10.875382 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.875403 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.875406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.875841 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.876077 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:10.876079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:10.876079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:10.876248 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:10.876737 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:10.876745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:10.877185 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 02:52:10.878219 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0619 02:52:10.878482 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":129,"threads_started":1}
I20260619 02:52:10.879875 7764 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36429
I20260619 02:52:10.880017 7764 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.880419 7764 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.881476 7764 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:36429 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:10.881703 7764 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.881871 7764 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:10.882174 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.868711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.26.98.62:36429)
0619 02:52:10.869141 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:10.869143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:10.869150 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:52:10.869178 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:10.869179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:10.869194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:10.869306 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:10.870600 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.871527 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.872812 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:10.873138 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:10.873214 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:10.875567 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:10.875569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:10.875569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:10.875579 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:10.878115 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.878629 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.878630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.878631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.878713 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:10.878966 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:10.878968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:10.878968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:10.879143 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:10.881065 (+ 1922us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:10.881137 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:52:10.881289 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":131,"threads_started":1}
I20260619 02:52:10.882477 7764 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:10.882571 7764 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
W20260619 02:52:10.883030 7764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:11.355625 7496 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:11.355842 7496 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:11.356618 7298 ts_manager.cc:194] Registered new tserver with Master: af93142dc84e4a60ba572a548edd7dc3 (127.26.98.1:36575)
I20260619 02:52:11.358136 7298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:35697
Jun 19 02:52:11 dist-test-slave-0f9d krb5kdc[6301](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837531, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:11.631860 7630 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:11.632091 7630 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:11.632457 7297 ts_manager.cc:194] Registered new tserver with Master: 05f1f38602d14b36ae94e806cfc34443 (127.26.98.2:34149)
I20260619 02:52:11.633314 7297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:47173
I20260619 02:52:11.883271 7764 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:11.883524 7764 heartbeater.cc:511] Master 127.26.98.62:36429 requested a full tablet report, sending...
I20260619 02:52:11.883831 7297 ts_manager.cc:194] Registered new tserver with Master: ed860430fd674137a7eab569e1fc5be2 (127.26.98.3:36905)
I20260619 02:52:11.884289 7297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:35223
I20260619 02:52:11.895299 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:11.909394 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.899318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.26.98.62:36429)
0619 02:52:11.899755 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.899757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.899767 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:11.899816 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.899817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.899832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.900782 (+ 950us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:11.900854 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.901715 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.903084 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.903436 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.903924 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.905361 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:11.905362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:11.905363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:11.905372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:11.907653 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.908278 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.908280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.908281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.908385 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.908642 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.908643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.908643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.908824 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:11.908946 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.909128 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.909252 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":84,"threads_started":1}
I20260619 02:52:11.912516 7297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:11.916024 7297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:11.933427 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.926024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36905 (local address 127.0.0.1:57068)
0619 02:52:11.927466 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.927479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:11.927556 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:11.928300 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:11.928304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:11.928308 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:11.928465 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.928468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.929843 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.929844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.931816 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.931819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.933028 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.933029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.933128 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.933145 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:52:11.933222 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:52:11.933303 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260619 02:52:11.933429 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.926521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.26.98.1:36575)
0619 02:52:11.927064 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.927070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.927397 (+ 327us) server_negotiation.cc:407] Connection header received
0619 02:52:11.928098 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.928101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.928122 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.928185 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:11.929496 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.930614 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.931932 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.932803 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.932884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.932981 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.933277 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.933333 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":83,"threads_started":1}
I20260619 02:52:11.933763 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.925625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36575 (local address 127.0.0.1:48424)
0619 02:52:11.927246 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.927302 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:11.927932 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:11.929322 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:11.929324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:11.929327 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:11.929393 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.929395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.930762 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.930763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.931816 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.931819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.933049 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.933050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.933128 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.933145 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:52:11.933222 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:52:11.933690 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1599,"mutex_wait_us":401,"thread_start_us":174,"threads_started":1}
I20260619 02:52:11.934301 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.926543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.26.98.3:36905)
0619 02:52:11.927171 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.927176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.927653 (+ 477us) server_negotiation.cc:407] Connection header received
0619 02:52:11.927704 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.927707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.927730 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.927816 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:11.928579 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.929744 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.931923 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.932802 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.932889 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.932989 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.933712 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.934208 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":75,"threads_started":1}
I20260619 02:52:11.934659 7417 tablet_service.cc:1467] Processing CreateTablet for tablet 24ae7ebcde214ce398652884f5797bc6 (DEFAULT_TABLE table=test-table [id=bac25c68d26949a0af43ca7a0f83472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:11.935181 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ae7ebcde214ce398652884f5797bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:11.935070 7698 tablet_service.cc:1467] Processing CreateTablet for tablet 24ae7ebcde214ce398652884f5797bc6 (DEFAULT_TABLE table=test-table [id=bac25c68d26949a0af43ca7a0f83472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:11.935709 7698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ae7ebcde214ce398652884f5797bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:11.941120 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.925618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:34149 (local address 127.0.0.1:47354)
0619 02:52:11.926284 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.929643 (+ 3359us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:11.929753 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:11.932422 (+ 2669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:11.932423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:11.932428 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:11.932493 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.932495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.935099 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.935101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.936503 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.936508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.937698 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.937699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.940821 (+ 3122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.940832 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:52:11.940922 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:52:11.941002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":175,"threads_started":1}
I20260619 02:52:11.941582 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.926031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.26.98.2:34149)
0619 02:52:11.926193 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.926201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.929838 (+ 3637us) server_negotiation.cc:407] Connection header received
0619 02:52:11.929896 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.929899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.929922 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.930006 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:11.932618 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.934336 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.936644 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.937482 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.937569 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.937655 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.941390 (+ 3735us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.941477 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260619 02:52:11.941990 7798 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Bootstrap starting.
I20260619 02:52:11.942757 7564 tablet_service.cc:1467] Processing CreateTablet for tablet 24ae7ebcde214ce398652884f5797bc6 (DEFAULT_TABLE table=test-table [id=bac25c68d26949a0af43ca7a0f83472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:11.943085 7798 tablet_bootstrap.cc:654] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:11.943503 7798 log.cc:824] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:11.943575 7564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ae7ebcde214ce398652884f5797bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:11.944101 7797 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Bootstrap starting.
I20260619 02:52:11.945398 7798 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: No bootstrap required, opened a new log
I20260619 02:52:11.945514 7798 ts_tablet_manager.cc:1397] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 02:52:11.945724 7797 tablet_bootstrap.cc:654] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:11.946161 7797 log.cc:824] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:11.947333 7797 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: No bootstrap required, opened a new log
I20260619 02:52:11.947450 7797 ts_tablet_manager.cc:1397] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:11.948861 7798 raft_consensus.cc:348] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.949116 7798 raft_consensus.cc:374] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:11.949162 7798 raft_consensus.cc:729] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed860430fd674137a7eab569e1fc5be2, State: Initialized, Role: FOLLOWER
I20260619 02:52:11.949327 7798 consensus_queue.cc:260] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.949748 7798 ts_tablet_manager.cc:1428] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 02:52:11.949918 7764 heartbeater.cc:503] Master 127.26.98.62:36429 was elected leader, sending a full tablet report...
I20260619 02:52:11.951198 7797 raft_consensus.cc:348] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.951452 7802 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Bootstrap starting.
I20260619 02:52:11.951519 7797 raft_consensus.cc:374] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:11.951560 7797 raft_consensus.cc:729] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93142dc84e4a60ba572a548edd7dc3, State: Initialized, Role: FOLLOWER
I20260619 02:52:11.951711 7797 consensus_queue.cc:260] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.952087 7797 ts_tablet_manager.cc:1428] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:52:11.952224 7496 heartbeater.cc:503] Master 127.26.98.62:36429 was elected leader, sending a full tablet report...
I20260619 02:52:11.952391 7802 tablet_bootstrap.cc:654] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:11.952744 7802 log.cc:824] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:11.953998 7802 tablet_bootstrap.cc:492] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: No bootstrap required, opened a new log
I20260619 02:52:11.954138 7802 ts_tablet_manager.cc:1397] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:11.956440 7802 raft_consensus.cc:348] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.956657 7802 raft_consensus.cc:374] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:11.956694 7802 raft_consensus.cc:729] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f1f38602d14b36ae94e806cfc34443, State: Initialized, Role: FOLLOWER
I20260619 02:52:11.956851 7802 consensus_queue.cc:260] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:11.957218 7802 ts_tablet_manager.cc:1428] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:52:11.957376 7630 heartbeater.cc:503] Master 127.26.98.62:36429 was elected leader, sending a full tablet report...
W20260619 02:52:12.084031 7497 tablet.cc:2369] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:12.096305 7631 tablet.cc:2369] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:12.113276 7765 tablet.cc:2369] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:52:12 dist-test-slave-0f9d krb5kdc[6301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837531, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for HTTP/127.26.98.20@KRBTEST.COM
I20260619 02:52:12.137660 7803 raft_consensus.cc:482] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:12.138167 7803 raft_consensus.cc:504] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:12.139765 7803 leader_election.cc:288] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f1f38602d14b36ae94e806cfc34443 (127.26.98.2:34149), ed860430fd674137a7eab569e1fc5be2 (127.26.98.3:36905)
I20260619 02:52:12.154390 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.140588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:52507 (local address 127.26.98.2:34149)
0619 02:52:12.140661 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:12.140663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:12.148148 (+ 7485us) server_negotiation.cc:407] Connection header received
0619 02:52:12.148288 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:12.148290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:12.148310 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:12.148345 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:12.148903 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.149680 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.152337 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.153779 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.153850 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:12.153892 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:12.154196 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 02:52:12.154269 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:52:12.154860 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.140336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:34149 (local address 127.26.98.1:52507)
0619 02:52:12.148041 (+ 7705us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.148062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:12.148199 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:12.148477 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:12.148479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:12.148482 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:12.148794 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:12.148800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.149848 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.149850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:12.152178 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:12.152189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.154009 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.154011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:12.154099 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:12.154104 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:12.154583 (+ 479us) client_negotiation.cc:240] Negotiation successful
0619 02:52:12.154736 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7658,"thread_start_us":205,"threads_started":1}
I20260619 02:52:12.155586 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ae7ebcde214ce398652884f5797bc6" candidate_uuid: "af93142dc84e4a60ba572a548edd7dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f1f38602d14b36ae94e806cfc34443" is_pre_election: true
I20260619 02:52:12.155891 7569 raft_consensus.cc:2454] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93142dc84e4a60ba572a548edd7dc3 in term 0.
I20260619 02:52:12.156387 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.141012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:55543 (local address 127.26.98.3:36905)
0619 02:52:12.141073 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:12.141076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:12.151260 (+ 10184us) server_negotiation.cc:407] Connection header received
0619 02:52:12.151425 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:12.151427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:12.151445 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:12.151479 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:12.151749 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.152709 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.154796 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.155520 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.155576 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:12.155590 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:12.156232 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 02:52:12.156304 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:52:12.156423 7384 leader_election.cc:302] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f1f38602d14b36ae94e806cfc34443, af93142dc84e4a60ba572a548edd7dc3; no voters:
I20260619 02:52:12.156833 7803 raft_consensus.cc:2781] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:12.156911 7803 raft_consensus.cc:482] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:12.156936 7803 raft_consensus.cc:3037] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:12.157172 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.140729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36905 (local address 127.26.98.1:55543)
0619 02:52:12.151037 (+ 10308us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.151058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:12.151320 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:12.151559 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:12.151560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:12.151563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:12.151664 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:12.151667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.152854 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.152855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:12.154388 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:12.154393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.156026 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.156028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:12.156120 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:12.156125 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:12.157004 (+ 879us) client_negotiation.cc:240] Negotiation successful
0619 02:52:12.157078 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10259,"thread_start_us":90,"threads_started":1}
I20260619 02:52:12.158159 7803 raft_consensus.cc:504] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:12.158370 7803 leader_election.cc:288] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [CANDIDATE]: Term 1 election: Requested vote from peers 05f1f38602d14b36ae94e806cfc34443 (127.26.98.2:34149), ed860430fd674137a7eab569e1fc5be2 (127.26.98.3:36905)
I20260619 02:52:12.158622 7718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ae7ebcde214ce398652884f5797bc6" candidate_uuid: "af93142dc84e4a60ba572a548edd7dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed860430fd674137a7eab569e1fc5be2" is_pre_election: true
I20260619 02:52:12.158864 7718 raft_consensus.cc:2454] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93142dc84e4a60ba572a548edd7dc3 in term 0.
I20260619 02:52:12.159806 7569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ae7ebcde214ce398652884f5797bc6" candidate_uuid: "af93142dc84e4a60ba572a548edd7dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f1f38602d14b36ae94e806cfc34443"
I20260619 02:52:12.159932 7569 raft_consensus.cc:3037] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:12.159149 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ae7ebcde214ce398652884f5797bc6" candidate_uuid: "af93142dc84e4a60ba572a548edd7dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed860430fd674137a7eab569e1fc5be2"
I20260619 02:52:12.161060 7713 raft_consensus.cc:3037] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:12.161396 7569 raft_consensus.cc:2454] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93142dc84e4a60ba572a548edd7dc3 in term 1.
I20260619 02:52:12.161691 7384 leader_election.cc:302] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f1f38602d14b36ae94e806cfc34443, af93142dc84e4a60ba572a548edd7dc3; no voters:
I20260619 02:52:12.161832 7803 raft_consensus.cc:2781] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:12.162057 7803 raft_consensus.cc:686] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [term 1 LEADER]: Becoming Leader. State: Replica: af93142dc84e4a60ba572a548edd7dc3, State: Running, Role: LEADER
I20260619 02:52:12.162233 7713 raft_consensus.cc:2454] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93142dc84e4a60ba572a548edd7dc3 in term 1.
I20260619 02:52:12.162233 7803 consensus_queue.cc:237] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } }
I20260619 02:52:12.163594 7298 catalog_manager.cc:5495] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to af93142dc84e4a60ba572a548edd7dc3 (127.26.98.1). New cstate: current_term: 1 leader_uuid: "af93142dc84e4a60ba572a548edd7dc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af93142dc84e4a60ba572a548edd7dc3" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 } health_report { overall_health: UNKNOWN } } }
I20260619 02:52:12.220036 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.216146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.26.98.1:36575)
0619 02:52:12.216242 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:12.216244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:12.216258 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:52:12.216329 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:12.216331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:12.216369 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:12.216483 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:12.216656 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.217591 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.218865 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:12.219220 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:12.219313 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:12.219534 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:12.219750 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:12.219861 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:12.219880 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:52:12.219924 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:52:12.223060 7713 raft_consensus.cc:1261] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 1 FOLLOWER]: Refusing update from remote peer af93142dc84e4a60ba572a548edd7dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:12.223335 7569 raft_consensus.cc:1261] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 1 FOLLOWER]: Refusing update from remote peer af93142dc84e4a60ba572a548edd7dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:12.223548 7808 consensus_queue.cc:1035] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed860430fd674137a7eab569e1fc5be2" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:12.223749 7803 consensus_queue.cc:1035] T 24ae7ebcde214ce398652884f5797bc6 P af93142dc84e4a60ba572a548edd7dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f1f38602d14b36ae94e806cfc34443" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:12.232583 7814 mvcc.cc:205] Tried to move back new op lower bound from 7298406531981459456 to 7298406531737407488. Current Snapshot: MvccSnapshot[applied={T|T < 7298406531981459456}]
I20260619 02:52:12.233227 7812 mvcc.cc:205] Tried to move back new op lower bound from 7298406531981459456 to 7298406531737407488. Current Snapshot: MvccSnapshot[applied={T|T < 7298406531981459456 or (T in {7298406531981459456})}]
I20260619 02:52:12.233623 7298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:12.233768 7298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:12.235195 7298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 562414d2563e4593a39a368a63493cfc: Sending DeleteTablet for 3 replicas of tablet 24ae7ebcde214ce398652884f5797bc6
I20260619 02:52:12.236070 7698 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ae7ebcde214ce398652884f5797bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:12 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:57068
I20260619 02:52:12.236207 7564 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ae7ebcde214ce398652884f5797bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:12 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47354
I20260619 02:52:12.236258 7417 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ae7ebcde214ce398652884f5797bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:12 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:48424
I20260619 02:52:12.237102 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7336
I20260619 02:52:12.239135 7822 tablet_replica.cc:330] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: stopping tablet replica
I20260619 02:52:12.239332 7822 raft_consensus.cc:2229] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:12.239483 7822 raft_consensus.cc:2258] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:12.240477 7822 ts_tablet_manager.cc:1905] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:12.244287 7822 ts_tablet_manager.cc:1918] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:12.244408 7822 log.cc:1196] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/24ae7ebcde214ce398652884f5797bc6
I20260619 02:52:12.244696 7822 ts_tablet_manager.cc:1939] T 24ae7ebcde214ce398652884f5797bc6 P 05f1f38602d14b36ae94e806cfc34443: Deleting consensus metadata
I20260619 02:52:12.245532 7283 catalog_manager.cc:4841] TS 05f1f38602d14b36ae94e806cfc34443 (127.26.98.2:34149): tablet 24ae7ebcde214ce398652884f5797bc6 (table test-table [id=bac25c68d26949a0af43ca7a0f83472b]) successfully deleted
I20260619 02:52:12.250283 7820 tablet_replica.cc:330] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: stopping tablet replica
I20260619 02:52:12.250546 7820 raft_consensus.cc:2229] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:12.250717 7820 raft_consensus.cc:2258] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:52:12.250720 7284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:36575 (error 108)
I20260619 02:52:12.251498 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7500
I20260619 02:52:12.251607 7820 ts_tablet_manager.cc:1905] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:12.253731 7820 ts_tablet_manager.cc:1918] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:12.253850 7820 log.cc:1196] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/24ae7ebcde214ce398652884f5797bc6
I20260619 02:52:12.254217 7820 ts_tablet_manager.cc:1939] T 24ae7ebcde214ce398652884f5797bc6 P ed860430fd674137a7eab569e1fc5be2: Deleting consensus metadata
I20260619 02:52:12.255017 7284 catalog_manager.cc:4841] TS ed860430fd674137a7eab569e1fc5be2 (127.26.98.3:36905): tablet 24ae7ebcde214ce398652884f5797bc6 (table test-table [id=bac25c68d26949a0af43ca7a0f83472b]) successfully deleted
I20260619 02:52:12.255978 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.255814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36575 (local address 127.0.0.1:48438)
0619 02:52:12.255887 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.255932 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:52:12.256143 7284 catalog_manager.cc:4584] TS af93142dc84e4a60ba572a548edd7dc3 (127.26.98.1:36575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24ae7ebcde214ce398652884f5797bc6: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36575: connect: Connection refused (error 111)
I20260619 02:52:12.259078 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7633
I20260619 02:52:12.264251 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7265
I20260619 02:52:12.271310 27016 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:52:12.617883 27016 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:52:12.618008 27016 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:52:12.965260 27016 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:52:12.965 UTC [6331] LOG: received smart shutdown request
2026-06-19 02:52:12.970 UTC [6331] LOG: background worker "logical replication launcher" (PID 6340) exited with exit code 1
2026-06-19 02:52:12.970 UTC [6335] LOG: shutting down
2026-06-19 02:52:13.259 UTC [6331] LOG: database system is shut down
2026-06-19T02:52:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (60571 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:13 dist-test-slave-0f9d krb5kdc[7832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:13 dist-test-slave-0f9d krb5kdc[7832](info): set up 2 sockets
Jun 19 02:52:13 dist-test-slave-0f9d krb5kdc[7832](info): commencing operation
krb5kdc: starting...
W20260619 02:52:14.383021 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:52:14.522593 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44253
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:14.621781 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:14.622102 7845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:14.622146 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:14.623695 7845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:14.623785 7845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:14.623808 7845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:14.623826 7845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:14.623840 7845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:14.625985 7845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:14.626371 7845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:14.626657 7845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:14.632248 7853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:14.632253 7850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:14.632380 7845 server_base.cc:1034] running on GCE node
W20260619 02:52:14.632253 7851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:14.632876 7845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:14.633252 7845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:14.634475 7845 hybrid_clock.cc:648] HybridClock initialized: now 1781837534634438 us; error 61 us; skew 500 ppm
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:14.638141 7845 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:14.638818 7845 webserver.cc:460] Webserver started at http://127.26.98.62:36953/ using document root <none> and password file <none>
I20260619 02:52:14.639153 7845 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:14.639215 7845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:14.639374 7845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:14.640753 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "d4b7041e39cc466b944371170d5cc224"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "c39ecb93d59c0322f00ad9c422c8ac7f"
server_key_iv: "90805f55228039540553489742db4752"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.641160 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "d4b7041e39cc466b944371170d5cc224"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "c39ecb93d59c0322f00ad9c422c8ac7f"
server_key_iv: "90805f55228039540553489742db4752"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.642764 7845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 02:52:14.644093 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:14.644487 7845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:14.644656 7845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "d4b7041e39cc466b944371170d5cc224"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "c39ecb93d59c0322f00ad9c422c8ac7f"
server_key_iv: "90805f55228039540553489742db4752"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.644764 7845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:14.666078 7845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:14.668730 7845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:14.668919 7845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:14.673746 7845 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:44253
I20260619 02:52:14.673797 7912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:44253 every 8 connection(s)
I20260619 02:52:14.674189 7845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:14.675266 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:14.677201 7913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:14.679507 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7845
I20260619 02:52:14.679621 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:52:14.679847 27016 external_mini_cluster.cc:1428] Setting key e9b4e1b9ffb62908da20f3ee08e28655
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.61@KRBTEST.COM; defaulting to no policy
I20260619 02:52:14.688802 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.678049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44253 (local address 127.0.0.1:48406)
0619 02:52:14.678314 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.678368 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:14.679037 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:14.679335 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:14.679338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:14.679723 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:14.679952 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.679957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.681462 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.681463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.682246 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.682249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.682778 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.682779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.682850 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.683530 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:14.683541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:14.685308 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:14.687213 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.687220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.687224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.687433 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.687652 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.687652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.687653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.687742 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.688124 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:14.688129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:14.688208 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 02:52:14.688298 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:52:14.688553 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":82,"thread_start_us":133,"threads_started":1}
I20260619 02:52:14.688800 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.678295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.26.98.62:44253)
0619 02:52:14.678435 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:14.678440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:14.678486 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:52:14.679212 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:14.679214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:14.679235 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:14.679325 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:14.680050 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.681366 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.682333 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.682700 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.682817 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.685418 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:14.685426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:14.685427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:14.685434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:14.687124 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.687506 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.687510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.687511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.687595 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.687793 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.687794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.687794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.688064 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:14.688141 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:14.688335 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:52:14.688495 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":109,"thread_start_us":46,"threads_started":1}
I20260619 02:52:14.689889 7913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:14.690292 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.690168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53666)
0619 02:52:14.690213 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.690264 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:14.690474 7862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
I20260619 02:52:14.691045 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.690953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53678)
0619 02:52:14.690983 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.691016 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 02:52:14.691147 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:44171: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
I20260619 02:52:14.710945 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:44171
--webserver_interface=127.26.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:14.740479 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } attempt: 1
I20260619 02:52:14.741191 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.740971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53684)
0619 02:52:14.741040 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.741081 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 02:52:14.741616 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.741518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53696)
0619 02:52:14.741559 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.741588 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:14.741735 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:44171: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
I20260619 02:52:14.809930 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } attempt: 2
I20260619 02:52:14.810657 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.810379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53710)
0619 02:52:14.810493 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.810595 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:52:14.811187 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.811088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53714)
0619 02:52:14.811132 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.811163 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:52:14.811299 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:44171: Network error: Client connection negotiation failed: client connection to 127.26.98.61:44171: connect: Connection refused (error 111)
W20260619 02:52:14.811888 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:14.812131 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:14.812160 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:14.813623 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:14.813668 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:14.813681 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:14.813694 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:14.813704 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:14.815809 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:14.816213 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:14.816496 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:14.822038 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:14.822127 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:14.822154 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:14.822196 7920 server_base.cc:1034] running on GCE node
I20260619 02:52:14.822633 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:14.823071 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:14.824271 7920 hybrid_clock.cc:648] HybridClock initialized: now 1781837534824255 us; error 49 us; skew 500 ppm
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:14.828215 7920 init.cc:377] Logged in from keytab as kudu/127.26.98.61@KRBTEST.COM (short username kudu)
I20260619 02:52:14.828890 7920 webserver.cc:460] Webserver started at http://127.26.98.61:39331/ using document root <none> and password file <none>
I20260619 02:52:14.829141 7920 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:14.829195 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:14.829311 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:14.830803 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data/instance:
uuid: "0a2caf67690745ac9cac23c90758c78a"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "5c231beccece0f844634e99aed4977e1"
server_key_iv: "81053f04a0163a8ec0ead14690456efc"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.831305 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal/instance:
uuid: "0a2caf67690745ac9cac23c90758c78a"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "5c231beccece0f844634e99aed4977e1"
server_key_iv: "81053f04a0163a8ec0ead14690456efc"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.832978 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:14.834151 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:14.834422 7920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:14.834549 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "0a2caf67690745ac9cac23c90758c78a"
format_stamp: "Formatted at 2026-06-19 02:52:14 on dist-test-slave-0f9d"
server_key: "5c231beccece0f844634e99aed4977e1"
server_key_iv: "81053f04a0163a8ec0ead14690456efc"
server_key_version: "encryptionkey@0"
I20260619 02:52:14.834656 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:14.856392 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:14.859673 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:14.859833 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:14.864667 7920 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.61:44171
I20260619 02:52:14.864739 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.61:44171 every 8 connection(s)
I20260619 02:52:14.865156 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
I20260619 02:52:14.866317 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:14.868060 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7920
I20260619 02:52:14.868166 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-1/wal/instance
I20260619 02:52:14.868384 27016 external_mini_cluster.cc:1428] Setting key 760931c6e4e425ae6c1ec3b0c7635dcb
I20260619 02:52:14.868479 7988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.60@KRBTEST.COM; defaulting to no policy
I20260619 02:52:14.879199 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.869371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.26.98.62:44253)
0619 02:52:14.869446 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:14.869448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:14.869660 (+ 212us) server_negotiation.cc:407] Connection header received
0619 02:52:14.870396 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:14.870397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:14.870414 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:14.870482 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:14.871413 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.872151 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.873248 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.873600 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.873685 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.876174 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:14.876176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:14.876176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:14.876185 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:14.877578 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.878028 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.878029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.878030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.878096 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.878381 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.878382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.878382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.878518 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:14.878593 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:14.878988 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 02:52:14.879080 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.60@KRBTEST.COM" created.
I20260619 02:52:14.879395 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.869320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44253 (local address 127.0.0.1:48422)
0619 02:52:14.869526 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.869575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:14.870252 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:14.870582 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:14.870587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:14.871038 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:14.871308 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.871313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.872271 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.872272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.873160 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.873163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.873688 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.873689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.873836 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.874435 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:14.874444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:14.876061 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:14.877666 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.877675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.877677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.877939 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.878188 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.878189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.878189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.878321 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.878582 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:14.878586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:14.878774 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 02:52:14.878931 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:52:14.879121 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260619 02:52:14.880142 7988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:52:14.891139 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.880629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53718)
0619 02:52:14.880711 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.880723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:14.880774 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:14.881267 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:14.881270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:14.881456 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:14.881585 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.881588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.883011 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.883013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.883574 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.883576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.884339 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.884342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.884527 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.885155 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:14.885161 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:14.887182 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:14.889780 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.889782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.889783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.889964 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.890237 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.890238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.890238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.890298 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.890664 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:14.890665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:14.890751 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 02:52:14.890876 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:52:14.891050 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:52:14.891265 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.880685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.26.98.61:44171)
0619 02:52:14.880941 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:14.880948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:14.880973 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:52:14.880998 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:14.881001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:14.881019 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:14.881110 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:14.881715 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.882821 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.883677 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.884181 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.884268 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.887471 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:14.887491 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:14.887501 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:14.887512 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:14.889641 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.890081 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.890085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.890085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.890158 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.890354 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.890354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.890355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.890559 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:14.890641 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:14.890967 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 02:52:14.891142 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":223,"thread_start_us":173,"threads_started":1}
I20260619 02:52:14.891990 7988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:14.892411 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.892277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51574)
0619 02:52:14.892321 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.892384 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:14.892571 7937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
I20260619 02:52:14.893085 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.892988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51588)
0619 02:52:14.893025 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.893050 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:52:14.893226 7988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
I20260619 02:52:14.898084 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:40395
--webserver_interface=127.26.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:14.902493 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } attempt: 3
Jun 19 02:52:14 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:52:14.912086 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.903029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53730)
0619 02:52:14.903090 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.903105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:14.903165 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:14.903371 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:14.903372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:14.903547 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:14.903648 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.903650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.904840 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.904842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.905432 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:14.905434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.906009 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.906010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:14.906098 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.906691 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:14.906698 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:14.908438 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:14.910595 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.910597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.910598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.910833 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.911203 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:14.911204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:14.911205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:14.911307 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:14.911704 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:14.911705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:14.911784 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 02:52:14.911866 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:52:14.911995 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 02:52:14.912106 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.903143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.26.98.61:44171)
0619 02:52:14.903213 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:14.903216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:14.903226 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:14.903264 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:14.903265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:14.903281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:14.903366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:14.903770 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.904587 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.905533 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:14.905876 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:14.905978 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:14.908566 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:14.908568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:14.908568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:14.908581 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:14.910450 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.910945 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.910947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.910948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.911020 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:14.911393 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:14.911394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:14.911394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:14.911591 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:14.911744 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:14.911900 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:52:14.912004 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:52:14.912606 7913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:14.913025 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.912916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51590)
0619 02:52:14.912964 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.912997 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:52:14.913348 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.913259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51596)
0619 02:52:14.913300 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.913325 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:52:14.913439 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
I20260619 02:52:14.942517 7988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 1
I20260619 02:52:14.943289 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.943116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51610)
0619 02:52:14.943192 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.943245 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:52:14.943723 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.943635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51626)
0619 02:52:14.943674 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.943702 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:52:14.943881 7988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
I20260619 02:52:14.944592 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 1
I20260619 02:52:14.945161 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.945037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51630)
0619 02:52:14.945092 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.945130 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:52:14.945561 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.945418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51638)
0619 02:52:14.945494 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.945530 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 02:52:14.945664 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
W20260619 02:52:14.999354 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:14.999644 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:14.999671 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:15.001045 7995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:15.001107 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:15.001121 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:15.001142 7995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:15.001153 7995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:15.003371 7995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44253,127.26.98.61:44171,127.26.98.60:40395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:15.003729 7995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:15.003993 7995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:15.009888 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:15.009878 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:15.009913 7995 server_base.cc:1034] running on GCE node
W20260619 02:52:15.009876 8003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:15.010540 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:15.010973 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:15.012120 7988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 2
I20260619 02:52:15.012243 7995 hybrid_clock.cc:648] HybridClock initialized: now 1781837535012211 us; error 40 us; skew 500 ppm
I20260619 02:52:15.012620 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.012481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51650)
0619 02:52:15.012537 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.012578 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:52:15.013738 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.013615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51654)
0619 02:52:15.013672 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.013702 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:52:15.013872 7988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837535, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:15.015725 7995 init.cc:377] Logged in from keytab as kudu/127.26.98.60@KRBTEST.COM (short username kudu)
I20260619 02:52:15.016369 7995 webserver.cc:460] Webserver started at http://127.26.98.60:33941/ using document root <none> and password file <none>
I20260619 02:52:15.016604 7995 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:15.016654 7995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:15.016788 7995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:15.018064 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data/instance:
uuid: "e231d6bd396f4c55967cd75ab9b06fe3"
format_stamp: "Formatted at 2026-06-19 02:52:15 on dist-test-slave-0f9d"
server_key: "3f06023da31a235c7f8d47414a19a855"
server_key_iv: "d17418bcd26b24f4d04d39a2a524b603"
server_key_version: "encryptionkey@0"
I20260619 02:52:15.018463 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal/instance:
uuid: "e231d6bd396f4c55967cd75ab9b06fe3"
format_stamp: "Formatted at 2026-06-19 02:52:15 on dist-test-slave-0f9d"
server_key: "3f06023da31a235c7f8d47414a19a855"
server_key_iv: "d17418bcd26b24f4d04d39a2a524b603"
server_key_version: "encryptionkey@0"
I20260619 02:52:15.020117 7995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:52:15.020826 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 2
I20260619 02:52:15.021095 8010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:15.021301 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.021159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51656)
0619 02:52:15.021221 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.021254 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:52:15.021588 7995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:15.021631 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.021539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51664)
0619 02:52:15.021579 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.021605 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:52:15.021703 7913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40395: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40395: connect: Connection refused (error 111)
I20260619 02:52:15.021703 7995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "e231d6bd396f4c55967cd75ab9b06fe3"
format_stamp: "Formatted at 2026-06-19 02:52:15 on dist-test-slave-0f9d"
server_key: "3f06023da31a235c7f8d47414a19a855"
server_key_iv: "d17418bcd26b24f4d04d39a2a524b603"
server_key_version: "encryptionkey@0"
I20260619 02:52:15.021826 7995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:15.033495 7995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:15.036396 7995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:15.036568 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:15.041687 7995 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.60:40395
I20260619 02:52:15.041728 8062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.60:40395 every 8 connection(s)
I20260619 02:52:15.042230 7995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
I20260619 02:52:15.043409 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:15.044934 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 7995
I20260619 02:52:15.045070 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837352638110-27016-0/minicluster-data/master-2/wal/instance
I20260619 02:52:15.045284 27016 external_mini_cluster.cc:1428] Setting key 152c28178930097655a76d6b6033827f
I20260619 02:52:15.045583 8063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837535, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:52:15.056902 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.046515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.26.98.62:44253)
0619 02:52:15.046614 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.046616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.046672 (+ 56us) server_negotiation.cc:407] Connection header received
0619 02:52:15.046960 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.046961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.046974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.047037 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.047854 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.048414 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.049609 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.049902 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.049960 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.052620 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.052622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.052623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.052634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.055591 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.055994 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.055995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.055996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.056059 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.056272 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.056273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.056273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.056425 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.056489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.056686 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.056790 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:52:15.057126 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.046408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44253 (local address 127.0.0.1:48424)
0619 02:52:15.046544 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.046592 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:15.046830 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:15.047071 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:15.047075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:15.047470 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:15.047741 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.047747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.048488 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.048490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.049518 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.049521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.049980 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.049981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.050112 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.050789 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:15.050802 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:15.052508 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:15.055673 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.055680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.055684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.055910 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.056116 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.056117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.056117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.056213 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.056486 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:15.056493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:15.056571 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 02:52:15.056659 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:52:15.056871 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260619 02:52:15.057513 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.045904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.26.98.60:40395)
0619 02:52:15.046138 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.046150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.046195 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:52:15.046821 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.046832 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.046868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.046994 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.047742 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.048887 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.049798 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.050222 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.050288 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.053080 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.053087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.053087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.053094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.055379 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.055919 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.055923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.055923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.056027 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.056257 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.056258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.056258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.056526 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.056680 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.056766 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.057409 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":171,"thread_start_us":124,"threads_started":1}
I20260619 02:52:15.057739 8063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837535, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:15.067101 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.058116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.26.98.61:44171)
0619 02:52:15.058185 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.058187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.058197 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:15.058294 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.058295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.058306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.058362 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.058678 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.059420 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.060234 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.060545 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.060630 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.063300 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.063303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.063303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.063313 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.065878 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.066260 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.066261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.066261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.066339 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.066526 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.066527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.066527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.066659 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.066717 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.066863 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.066985 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:52:15.067101 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.058036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:44171 (local address 127.0.0.1:53738)
0619 02:52:15.058161 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.058173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:15.058209 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:15.058365 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:15.058366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:15.058531 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:15.058606 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.058608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.059489 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.059490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.060149 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.060151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.060643 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.060644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.060746 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.061279 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:15.061283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:15.063116 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:15.065993 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.065995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.065995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.066180 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.066415 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.066416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.066416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.066468 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.066725 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:15.066726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:15.066776 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:52:15.066830 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:52:15.066985 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 02:52:15.067318 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.058420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.26.98.62:44253)
0619 02:52:15.058462 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.058464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.058504 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:52:15.058552 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.058552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.058563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.058623 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.059214 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.059816 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.060748 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.061022 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.061074 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.063948 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.063952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.063953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.063968 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.065831 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.066257 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.066258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.066259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.066339 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.066610 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.066611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.066612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.066755 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.067008 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.067047 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.067142 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:52:15.068110 8063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837535, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:52:15.077304 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.068503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.26.98.60:40395)
0619 02:52:15.068546 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.068548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.068600 (+ 52us) server_negotiation.cc:407] Connection header received
0619 02:52:15.068676 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.068676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.068685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.068725 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.068993 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.069770 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.070828 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.071211 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.071288 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.074089 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.074091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.074091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.074102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.075766 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.076151 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.076152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.076153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.076227 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.076492 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.076493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.076493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.076674 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.076738 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.077116 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.077205 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:52:15.077404 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.068404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51690)
0619 02:52:15.068551 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.068562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:15.068620 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:15.068732 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:15.068733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:15.068877 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:15.068925 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.068926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.069875 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.069876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.070745 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.070747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.071295 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.071297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.071387 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.072063 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:15.072073 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:15.073947 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:15.075889 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.075891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.075891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.076079 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.076296 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.076297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.076297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.076345 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.076806 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:15.076809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:15.076975 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 02:52:15.077081 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:52:15.077272 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":62}
I20260619 02:52:15.078361 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.068103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.26.98.61:44171)
0619 02:52:15.068136 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.068138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.068147 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:15.068268 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.068269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.068280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.068326 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.068953 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.069770 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.070829 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.071182 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.071261 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.074953 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.074956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.074956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.074966 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.076604 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.077322 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.077323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.077324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.077417 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.077678 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.077679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.077679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.077805 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.078048 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.078156 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.078249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:52:15.079566 8063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: Bootstrap starting.
I20260619 02:52:15.080145 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:52:15.080706 8063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:15.081079 8063 log.cc:824] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:15.082172 8063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: No bootstrap required, opened a new log
I20260619 02:52:15.085465 8063 raft_consensus.cc:348] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.085738 8063 raft_consensus.cc:374] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:15.085777 8063 raft_consensus.cc:729] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e231d6bd396f4c55967cd75ab9b06fe3, State: Initialized, Role: FOLLOWER
I20260619 02:52:15.085996 8063 consensus_queue.cc:260] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.086489 8063 sys_catalog.cc:564] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:15.087101 8076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.087231 8076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:52:15.089787 8087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:15.089850 8087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:15.091394 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.082241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.26.98.62:44253)
0619 02:52:15.082539 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.082542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.082555 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:52:15.082591 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.082592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.082606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.082683 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.083929 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.084617 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.085871 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.086176 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.086223 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.088294 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.088296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.088297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.088307 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.089999 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.090431 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.090432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.090433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.090510 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.090712 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.090712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.090713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.090906 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.091008 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.091192 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.091283 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260619 02:52:15.092864 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.082114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.26.98.61:44171)
0619 02:52:15.082210 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.082213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.082861 (+ 648us) server_negotiation.cc:407] Connection header received
0619 02:52:15.082980 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.082981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.082994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.083052 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.083642 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.084401 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.086241 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.086530 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.086577 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.089612 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.089615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.089615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.089625 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.091407 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.091928 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.091929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.091930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.091996 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.092192 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.092193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.092193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.092383 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.092506 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.092671 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.092764 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:52:15.089100 7995 master_runner.cc:186] Error getting master registration for 127.26.98.62:44253: OK, instance_id { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" instance_seqno: 1781837534672456 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:52:15.093474 7995 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:52:15.095068 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.082815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.26.98.60:40395)
0619 02:52:15.082860 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.082862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.084058 (+ 1196us) server_negotiation.cc:407] Connection header received
0619 02:52:15.084166 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.084168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.084181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.084215 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.085293 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.085989 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.087595 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.088082 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.088167 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.091998 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.092000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.092000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.092010 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.093897 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.094253 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.094255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.094255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.094311 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.094483 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.094483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.094484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.094634 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.094726 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.094858 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.094973 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:52:15.105172 7988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 3
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:52:15.115321 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.105800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51718)
0619 02:52:15.105901 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.105919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:15.105987 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:15.106240 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:15.106241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:15.106489 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:15.106570 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.106572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.107727 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.107728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.108600 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.108603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.109099 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.109100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.109176 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.109748 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:15.109755 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:15.111557 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:15.113808 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.113810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.113811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.114029 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.114329 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.114331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.114331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.114458 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.114832 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:15.114833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:15.114949 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:52:15.115027 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:52:15.115208 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:52:15.115321 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.105943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.26.98.60:40395)
0619 02:52:15.106062 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.106064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.106082 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:52:15.106116 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.106117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.106130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.106222 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.106735 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.107595 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.108696 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.109004 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.109087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.111709 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.111712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.111713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.111723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.113658 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.114108 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.114109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.114110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.114178 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.114551 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.114552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.114552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.114727 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.114808 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.115089 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.115201 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:52:15.117826 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: Bootstrap starting.
I20260619 02:52:15.118848 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:15.119292 7988 log.cc:824] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:15.120476 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: No bootstrap required, opened a new log
I20260619 02:52:15.120862 7913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } attempt: 3
I20260619 02:52:15.123392 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.123652 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:15.123695 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2caf67690745ac9cac23c90758c78a, State: Initialized, Role: FOLLOWER
I20260619 02:52:15.123906 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.124404 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:15.125645 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
Jun 19 02:52:15 dist-test-slave-0f9d krb5kdc[7832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837534, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:52:15.126456 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:52:15.128791 8101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:15.128995 8101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:15.127115 7920 master_runner.cc:186] Error getting master registration for 127.26.98.62:44253: OK, instance_id { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" instance_seqno: 1781837534672456 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:52:15.130016 7920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:52:15.132066 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.121345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40395 (local address 127.0.0.1:51726)
0619 02:52:15.121427 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.121442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:15.121497 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:15.121669 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:15.121670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:15.121865 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:15.121923 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.121926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.122650 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.122652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.123440 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:15.123451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.123948 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.123950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:15.124010 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.124746 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:15.124754 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:15.127003 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:15.130454 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.130457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.130458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.130795 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.131099 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:15.131103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:15.131103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:15.131181 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:15.131638 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:15.131639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:15.131708 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 02:52:15.131797 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:52:15.131949 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:52:15.132344 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.121426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.26.98.60:40395)
0619 02:52:15.121479 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:15.121481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:15.121498 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:52:15.121578 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:15.121579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:15.121591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:15.121645 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:15.122017 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.122544 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.123656 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:15.123871 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:15.123940 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:15.127355 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:15.127357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:15.127358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:15.127369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:15.130177 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.130945 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.130947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.130948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.131027 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:15.131313 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:15.131314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:15.131315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:15.131535 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:15.131628 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:15.131966 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 02:52:15.132240 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:52:15.134687 7913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: Bootstrap starting.
I20260619 02:52:15.135833 7913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:15.136260 7913 log.cc:824] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:15.137579 7913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: No bootstrap required, opened a new log
I20260619 02:52:15.140715 7913 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.141011 7913 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:15.141052 7913 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b7041e39cc466b944371170d5cc224, State: Initialized, Role: FOLLOWER
I20260619 02:52:15.141255 7913 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.141808 7913 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:15.141768 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.142022 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:52:15.144194 8115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:15.144398 8115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:15.148974 7845 master_runner.cc:263] No leader master found from master d4b7041e39cc466b944371170d5cc224
I20260619 02:52:15.149113 7845 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:52:15.170085 7866 tablet.cc:2369] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:15.251592 8090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:15.251776 8090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.252089 8090 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b7041e39cc466b944371170d5cc224 (127.26.98.62:44253), e231d6bd396f4c55967cd75ab9b06fe3 (127.26.98.60:40395)
I20260619 02:52:15.252578 7887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a2caf67690745ac9cac23c90758c78a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b7041e39cc466b944371170d5cc224" is_pre_election: true
I20260619 02:52:15.252789 7887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2caf67690745ac9cac23c90758c78a in term 0.
I20260619 02:52:15.253046 7936 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2caf67690745ac9cac23c90758c78a, d4b7041e39cc466b944371170d5cc224; no voters:
I20260619 02:52:15.253211 8090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:15.253260 8090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:15.253295 8090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:15.254707 8090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.255084 7887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a2caf67690745ac9cac23c90758c78a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b7041e39cc466b944371170d5cc224"
I20260619 02:52:15.255191 7887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:15.256242 7887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2caf67690745ac9cac23c90758c78a in term 1.
I20260619 02:52:15.256781 8037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a2caf67690745ac9cac23c90758c78a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" is_pre_election: true
I20260619 02:52:15.256997 8037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2caf67690745ac9cac23c90758c78a in term 0.
I20260619 02:52:15.257143 8036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a2caf67690745ac9cac23c90758c78a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231d6bd396f4c55967cd75ab9b06fe3"
I20260619 02:52:15.257227 8036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:15.258205 8036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2caf67690745ac9cac23c90758c78a in term 1.
I20260619 02:52:15.254864 8090 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [CANDIDATE]: Term 1 election: Requested vote from peers d4b7041e39cc466b944371170d5cc224 (127.26.98.62:44253), e231d6bd396f4c55967cd75ab9b06fe3 (127.26.98.60:40395)
I20260619 02:52:15.256454 7936 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2caf67690745ac9cac23c90758c78a, d4b7041e39cc466b944371170d5cc224; no voters:
I20260619 02:52:15.258795 8090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:15.258996 8090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [term 1 LEADER]: Becoming Leader. State: Replica: 0a2caf67690745ac9cac23c90758c78a, State: Running, Role: LEADER
I20260619 02:52:15.259109 8090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } }
I20260619 02:52:15.259585 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2caf67690745ac9cac23c90758c78a. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.259671 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:15.260991 8119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:15.261241 8119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:15.263855 8036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [term 1 FOLLOWER]: Refusing update from remote peer 0a2caf67690745ac9cac23c90758c78a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:15.264267 8090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:15.265986 8076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2caf67690745ac9cac23c90758c78a. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.266129 8076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:15.266646 7887 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [term 1 FOLLOWER]: Refusing update from remote peer 0a2caf67690745ac9cac23c90758c78a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:15.267061 8090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:15.268980 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2caf67690745ac9cac23c90758c78a. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.269111 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:15.271517 8117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.271701 8117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:15.272518 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.272732 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:15.273099 8120 mvcc.cc:205] Tried to move back new op lower bound from 7298406544437805056 to 7298406544422076416. Current Snapshot: MvccSnapshot[applied={T|T < 7298406544437805056}]
I20260619 02:52:15.273478 8123 mvcc.cc:205] Tried to move back new op lower bound from 7298406544437805056 to 7298406544422076416. Current Snapshot: MvccSnapshot[applied={T|T < 7298406544437805056}]
I20260619 02:52:15.274052 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.274186 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:15.274935 8076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a2caf67690745ac9cac23c90758c78a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b7041e39cc466b944371170d5cc224" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44253 } } peers { permanent_uuid: "0a2caf67690745ac9cac23c90758c78a" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 44171 } } peers { permanent_uuid: "e231d6bd396f4c55967cd75ab9b06fe3" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40395 } } }
I20260619 02:52:15.275182 8076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:15.276469 8121 mvcc.cc:205] Tried to move back new op lower bound from 7298406544437805056 to 7298406544422076416. Current Snapshot: MvccSnapshot[applied={T|T < 7298406544437805056}]
I20260619 02:52:15.277684 8119 catalog_manager.cc:1347] Generated new cluster ID: 3d10762dc01843d7b405ca2ace79b88b
I20260619 02:52:15.277767 8119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:15.290835 8119 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:15.291790 8119 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:15.306671 8119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2caf67690745ac9cac23c90758c78a: Generated new TSK 0
I20260619 02:52:15.307078 8119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:52:16.090562 8087 catalog_manager.cc:1259] Loaded cluster ID: 3d10762dc01843d7b405ca2ace79b88b
I20260619 02:52:16.090674 8087 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: loading cluster ID for follower catalog manager: success
I20260619 02:52:16.092077 8087 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: acquiring CA information for follower catalog manager: success
I20260619 02:52:16.092446 8087 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e231d6bd396f4c55967cd75ab9b06fe3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:52:16.145020 8115 catalog_manager.cc:1259] Loaded cluster ID: 3d10762dc01843d7b405ca2ace79b88b
I20260619 02:52:16.145126 8115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: loading cluster ID for follower catalog manager: success
I20260619 02:52:16.146047 8115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: acquiring CA information for follower catalog manager: success
I20260619 02:52:16.146409 8115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4b7041e39cc466b944371170d5cc224: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:52:16.375006 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7845
I20260619 02:52:16.380702 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7920
I20260619 02:52:16.386673 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 7995
2026-06-19T02:52:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3023 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:16 dist-test-slave-0f9d krb5kdc[8135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:16 dist-test-slave-0f9d krb5kdc[8135](info): set up 2 sockets
Jun 19 02:52:16 dist-test-slave-0f9d krb5kdc[8135](info): commencing operation
krb5kdc: starting...
W20260619 02:52:17.407480 27016 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:52:17.547530 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:42033
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:42033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:17.646811 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:17.647159 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:17.647197 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:17.648613 8148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:17.648679 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:17.648700 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:17.648718 8148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:17.648733 8148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:17.651578 8148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:42033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:17.652022 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:17.652335 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:17.658625 8154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:17.658638 8153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:17.658756 8156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:17.658820 8148 server_base.cc:1034] running on GCE node
I20260619 02:52:17.659354 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:17.659721 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:17.660902 8148 hybrid_clock.cc:648] HybridClock initialized: now 1781837537660887 us; error 42 us; skew 500 ppm
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837537, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:17.664413 8148 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:17.665056 8148 webserver.cc:460] Webserver started at http://127.26.98.62:42815/ using document root <none> and password file <none>
I20260619 02:52:17.665302 8148 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:17.665354 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:17.665472 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:17.666805 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "6c492ad4d5354b619eca855b0a1cdd2a"
format_stamp: "Formatted at 2026-06-19 02:52:17 on dist-test-slave-0f9d"
server_key: "88ca60ab2842c67dc841680162b678f0"
server_key_iv: "c1bfe75756c40624f7f555dd9c0b92e1"
server_key_version: "encryptionkey@0"
I20260619 02:52:17.667327 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "6c492ad4d5354b619eca855b0a1cdd2a"
format_stamp: "Formatted at 2026-06-19 02:52:17 on dist-test-slave-0f9d"
server_key: "88ca60ab2842c67dc841680162b678f0"
server_key_iv: "c1bfe75756c40624f7f555dd9c0b92e1"
server_key_version: "encryptionkey@0"
I20260619 02:52:17.670632 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:17.671880 8163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:17.672324 8148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:17.672464 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "6c492ad4d5354b619eca855b0a1cdd2a"
format_stamp: "Formatted at 2026-06-19 02:52:17 on dist-test-slave-0f9d"
server_key: "88ca60ab2842c67dc841680162b678f0"
server_key_iv: "c1bfe75756c40624f7f555dd9c0b92e1"
server_key_version: "encryptionkey@0"
I20260619 02:52:17.672600 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:17.691363 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:17.694139 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:17.694324 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:17.700085 8148 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:42033
I20260619 02:52:17.700206 8215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:42033 every 8 connection(s)
I20260619 02:52:17.700654 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:17.701894 8216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:17.705318 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: Bootstrap starting.
I20260619 02:52:17.705639 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8148
I20260619 02:52:17.705801 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:52:17.706022 27016 external_mini_cluster.cc:1428] Setting key a2e04a810268ec57e26b422b489c52da
I20260619 02:52:17.706272 8216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:17.706717 8216 log.cc:824] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:17.707772 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: No bootstrap required, opened a new log
I20260619 02:52:17.710675 8216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } }
I20260619 02:52:17.711000 8216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:17.711061 8216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c492ad4d5354b619eca855b0a1cdd2a, State: Initialized, Role: FOLLOWER
I20260619 02:52:17.711247 8216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } }
I20260619 02:52:17.711336 8216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:17.711378 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:17.711436 8216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:17.712349 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } }
I20260619 02:52:17.712532 8216 leader_election.cc:302] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c492ad4d5354b619eca855b0a1cdd2a; no voters:
I20260619 02:52:17.712779 8216 leader_election.cc:288] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:17.712906 8221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:17.713114 8221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [term 1 LEADER]: Becoming Leader. State: Replica: 6c492ad4d5354b619eca855b0a1cdd2a, State: Running, Role: LEADER
I20260619 02:52:17.713152 8216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:17.713223 8221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } }
I20260619 02:52:17.713644 8221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c492ad4d5354b619eca855b0a1cdd2a. Latest consensus state: current_term: 1 leader_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } } }
I20260619 02:52:17.713783 8221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:17.713653 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c492ad4d5354b619eca855b0a1cdd2a" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 42033 } } }
I20260619 02:52:17.714967 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a [sys.catalog]: This master's current role is: LEADER
W20260619 02:52:17.715595 8236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:17.715669 8236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:17.715854 8227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:17.716110 8227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:17.718325 8227 catalog_manager.cc:1347] Generated new cluster ID: 9d77d73fc5834c37aa07afcbd28a6460
I20260619 02:52:17.718395 8227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:17.721441 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:17.706706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.26.98.62:42033)
0619 02:52:17.707056 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:17.707069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:17.707113 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:52:17.707959 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:17.707975 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:17.708068 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:17.708286 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:17.709248 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:17.710825 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:17.712134 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:17.712884 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:17.713032 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:17.716560 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:17.716566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:17.716567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:17.716585 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:17.718926 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:17.719424 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:17.719427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:17.719428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:17.719551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:17.719721 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:17.719722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:17.719722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:17.720074 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:17.720237 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:17.720774 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 02:52:17.720954 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260619 02:52:17.722997 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:52:17.728325 8227 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:17.728964 8227 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:17.740882 8227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c492ad4d5354b619eca855b0a1cdd2a: Generated new TSK 0
I20260619 02:52:17.741156 8227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:52:17.778337 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8148
2026-06-19T02:52:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1390 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8243](info): set up 2 sockets
Jun 19 02:52:17 dist-test-slave-0f9d krb5kdc[8243](info): commencing operation
krb5kdc: starting...
W20260619 02:52:18.779198 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:52:18 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:52:18.920070 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:19.024202 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:19.024523 8256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:19.024564 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:19.026134 8256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:19.026216 8256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:19.026235 8256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:19.026253 8256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:19.026270 8256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:19.028376 8256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:19.029066 8256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:19.029431 8256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:19.035526 8264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.035532 8261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.035538 8262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:19.035589 8256 server_base.cc:1034] running on GCE node
I20260619 02:52:19.036049 8256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:19.036450 8256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:19.037699 8256 hybrid_clock.cc:648] HybridClock initialized: now 1781837539037655 us; error 45 us; skew 500 ppm
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:19.045837 8256 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:19.046540 8256 webserver.cc:460] Webserver started at http://127.26.98.62:43259/ using document root <none> and password file <none>
I20260619 02:52:19.046816 8256 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:19.046873 8256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:19.047050 8256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:19.048321 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "0a29b8a3f271444abd131d88328d1d0f"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "b064016d3c1d4d79e40b4eef5d4faf02"
server_key_iv: "4ac99d0ad4b79216d9ea1c729b979c69"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.048707 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "0a29b8a3f271444abd131d88328d1d0f"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "b064016d3c1d4d79e40b4eef5d4faf02"
server_key_iv: "4ac99d0ad4b79216d9ea1c729b979c69"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.050349 8256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:52:19.051592 8271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.052062 8256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:19.052242 8256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0a29b8a3f271444abd131d88328d1d0f"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "b064016d3c1d4d79e40b4eef5d4faf02"
server_key_iv: "4ac99d0ad4b79216d9ea1c729b979c69"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.052353 8256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:19.062181 8256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:19.065379 8256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:19.065680 8256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:19.071547 8256 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35015
I20260619 02:52:19.071662 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35015 every 8 connection(s)
I20260619 02:52:19.072350 8256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:19.073608 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:19.076880 8324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap starting.
I20260619 02:52:19.077338 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8256
I20260619 02:52:19.077482 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:52:19.077682 8324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:19.077836 27016 external_mini_cluster.cc:1428] Setting key 9a4e2b4716376753ce2164c577658528
I20260619 02:52:19.078015 8324 log.cc:824] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:19.078986 8324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: No bootstrap required, opened a new log
I20260619 02:52:19.081243 8324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:19.081435 8324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:19.081468 8324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Initialized, Role: FOLLOWER
I20260619 02:52:19.081804 8324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:19.082047 8324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:19.082098 8324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:19.082263 8324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:19.083453 8324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:19.083654 8324 leader_election.cc:302] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a29b8a3f271444abd131d88328d1d0f; no voters:
I20260619 02:52:19.083950 8324 leader_election.cc:288] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:19.083998 8329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:19.084138 8329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Running, Role: LEADER
I20260619 02:52:19.084345 8324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:19.084345 8329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:19.084972 8330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:19.085031 8331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a29b8a3f271444abd131d88328d1d0f. Latest consensus state: current_term: 1 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:19.085165 8330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:19.085210 8331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
W20260619 02:52:19.086613 8344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:19.086675 8344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:19.086997 8338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:19.087276 8338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:19.090566 8338 catalog_manager.cc:1347] Generated new cluster ID: 2f7d30b8814e473da31621672f3b2692
I20260619 02:52:19.090636 8338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:19.091495 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.078521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.26.98.62:35015)
0619 02:52:19.078694 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.078705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.078737 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:52:19.079385 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.079394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.079455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.079608 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:19.080287 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.081396 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.082374 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.082946 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.083074 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.086678 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:19.086684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:19.086684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:19.086702 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:19.089040 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.089687 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.089693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.089694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.089851 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.090168 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.090169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.090170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.090578 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:19.090805 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.091058 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.091237 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260619 02:52:19.104411 8338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:19.105403 8338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:19.119503 8338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Generated new TSK 0
I20260619 02:52:19.119858 8338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:52:19.160746 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:19.267791 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:19.268100 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:19.268137 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:19.269955 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:19.270052 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:19.270102 8350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:19.272464 8350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:19.272863 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:19.273166 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:19.279486 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.279546 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.279621 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:19.279798 8350 server_base.cc:1034] running on GCE node
I20260619 02:52:19.280051 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:19.280369 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:19.281556 8350 hybrid_clock.cc:648] HybridClock initialized: now 1781837539281535 us; error 42 us; skew 500 ppm
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:19.284709 8350 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:19.285352 8350 webserver.cc:460] Webserver started at http://127.26.98.1:33903/ using document root <none> and password file <none>
I20260619 02:52:19.285617 8350 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:19.285672 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:19.285847 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:19.287431 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "93e34039cacae4cae02c624001ab6d5d"
server_key_iv: "70dd7ba2696b67069290c88f20261ad6"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.287916 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "93e34039cacae4cae02c624001ab6d5d"
server_key_iv: "70dd7ba2696b67069290c88f20261ad6"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.289906 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:19.291354 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.291651 8350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:19.291821 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "93e34039cacae4cae02c624001ab6d5d"
server_key_iv: "70dd7ba2696b67069290c88f20261ad6"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.291935 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:19.312450 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:19.315137 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:19.315320 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:19.315594 8350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:19.315958 8350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:19.315992 8350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.316027 8350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:19.316049 8350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.323715 8350 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:42079
I20260619 02:52:19.323800 8478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:42079 every 8 connection(s)
I20260619 02:52:19.324167 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:19.328644 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8350
I20260619 02:52:19.328776 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:52:19.329005 27016 external_mini_cluster.cc:1428] Setting key b9c96a13e0e0cee0ca06486a2b814777
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:19.336176 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.325081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:42299 (local address 127.26.98.62:35015)
0619 02:52:19.325176 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.325179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.325581 (+ 402us) server_negotiation.cc:407] Connection header received
0619 02:52:19.326120 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.326125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.326140 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.326225 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:19.327327 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.327956 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.329214 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.329476 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.329575 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.332422 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:19.332425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:19.332425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:19.332436 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:19.334110 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.334872 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.334906 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.334906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.334979 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.335316 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.335318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.335318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.335466 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:19.335536 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.335978 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.336078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:52:19.336460 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.325032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:42299)
0619 02:52:19.325315 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.325368 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.325992 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.326284 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.326288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.326958 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:19.327184 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.327190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.328117 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.328118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.329107 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.329111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.329634 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.329636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.329874 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.330574 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:19.330585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:19.332277 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:19.334273 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.334298 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.334303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.334736 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.335047 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.335048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.335048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.335222 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.335537 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:19.335545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:19.335775 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.335928 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.336163 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":152,"threads_started":1}
I20260619 02:52:19.337107 8479 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:19.337234 8479 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:19.337486 8479 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:19.338135 8288 ts_manager.cc:194] Registered new tserver with Master: d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:52:19.338842 8288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:42299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:52:19.361775 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:19.460825 8484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:19.461236 8484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:19.461292 8484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:19.463310 8484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:19.463531 8484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:19.463622 8484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:19.466431 8484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:19.466814 8484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:19.467191 8484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:19.474009 8489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.474009 8492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.474120 8490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:19.474664 8484 server_base.cc:1034] running on GCE node
I20260619 02:52:19.475026 8484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:19.475348 8484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:19.476577 8484 hybrid_clock.cc:648] HybridClock initialized: now 1781837539476535 us; error 63 us; skew 500 ppm
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:19.479549 8484 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:19.480094 8484 webserver.cc:460] Webserver started at http://127.26.98.2:38629/ using document root <none> and password file <none>
I20260619 02:52:19.480434 8484 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:19.480500 8484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:19.480634 8484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:19.482002 8484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "4749dc9ee18b4dafae4195f902057137"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "5c08a54f3f41008812fa8bc1968811cb"
server_key_iv: "1452d3c3589741ea15c3c9138915fb67"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.482427 8484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "4749dc9ee18b4dafae4195f902057137"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "5c08a54f3f41008812fa8bc1968811cb"
server_key_iv: "1452d3c3589741ea15c3c9138915fb67"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.484148 8484 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:52:19.485240 8499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.485651 8484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:19.485756 8484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "4749dc9ee18b4dafae4195f902057137"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "5c08a54f3f41008812fa8bc1968811cb"
server_key_iv: "1452d3c3589741ea15c3c9138915fb67"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.485860 8484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:19.519315 8484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:19.522682 8484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:19.522928 8484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:19.523243 8484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:19.523667 8484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:19.523720 8484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.523747 8484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:19.523759 8484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.531980 8484 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:35043
I20260619 02:52:19.532042 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:35043 every 8 connection(s)
I20260619 02:52:19.532788 8484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:19.539106 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8484
I20260619 02:52:19.539268 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:52:19.539474 27016 external_mini_cluster.cc:1428] Setting key 76228f65156b2aa238d0a1ebbca23be1
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:19.544929 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.534144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:56457 (local address 127.26.98.62:35015)
0619 02:52:19.534270 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.534273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.534350 (+ 77us) server_negotiation.cc:407] Connection header received
0619 02:52:19.535127 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.535129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.535147 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.535208 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:19.536520 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.537221 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.538679 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.539143 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.539232 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.541956 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:19.541958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:19.541958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:19.541970 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:19.543412 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.543892 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.543894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.543894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.543952 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.544201 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.544201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.544202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.544342 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:19.544404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.544714 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.544818 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 02:52:19.545163 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.533991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:56457)
0619 02:52:19.534215 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.534266 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.534940 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.535320 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.535325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.536068 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:19.536342 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.536359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.537358 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.537360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.538534 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.538538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.539277 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.539279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.539470 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.540117 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:19.540129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:19.541834 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:19.543508 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.543520 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.543524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.543817 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.544010 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.544011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.544011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.544143 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.544398 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:19.544405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:19.544582 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.544684 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.544899 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260619 02:52:19.545969 8613 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:19.546101 8613 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:19.546350 8613 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:19.546813 8288 ts_manager.cc:194] Registered new tserver with Master: 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043)
I20260619 02:52:19.547441 8288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:56457
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:52:19.569921 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:19.667088 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:19.667493 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:19.667634 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:19.669961 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:19.670027 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:19.670141 8618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:19.672498 8618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:19.672859 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:19.673115 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:19.679345 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.679364 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:19.679349 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:19.679699 8618 server_base.cc:1034] running on GCE node
I20260619 02:52:19.679986 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:19.680356 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:19.681547 8618 hybrid_clock.cc:648] HybridClock initialized: now 1781837539681509 us; error 51 us; skew 500 ppm
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:19.684890 8618 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:19.685563 8618 webserver.cc:460] Webserver started at http://127.26.98.3:43761/ using document root <none> and password file <none>
I20260619 02:52:19.685858 8618 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:19.685916 8618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:19.686057 8618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:19.687455 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "5ad7dec64be141888303b1f697a54de6"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "82ae602b93437208028e1fb91dc4ab7c"
server_key_iv: "6137f995e0e27284a3b014d1a7d89675"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.687887 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "5ad7dec64be141888303b1f697a54de6"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "82ae602b93437208028e1fb91dc4ab7c"
server_key_iv: "6137f995e0e27284a3b014d1a7d89675"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.689724 8618 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:52:19.690589 8633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.690764 8618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:19.690868 8618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "5ad7dec64be141888303b1f697a54de6"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "82ae602b93437208028e1fb91dc4ab7c"
server_key_iv: "6137f995e0e27284a3b014d1a7d89675"
server_key_version: "encryptionkey@0"
I20260619 02:52:19.690991 8618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:19.702946 8618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:19.705739 8618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:19.705921 8618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:19.706195 8618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:19.706550 8618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:19.706590 8618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.706617 8618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:19.706629 8618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:19.714237 8618 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:41135
I20260619 02:52:19.714340 8746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:41135 every 8 connection(s)
I20260619 02:52:19.714920 8618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:19.717334 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8618
I20260619 02:52:19.717461 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:52:19.717919 27016 external_mini_cluster.cc:1428] Setting key a8844a01b969582228a4359337ee8156
Jun 19 02:52:19 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837539, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:19.727636 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.715931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:51219 (local address 127.26.98.62:35015)
0619 02:52:19.716028 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.716031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.716263 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:52:19.716911 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.716913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.716926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.716995 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:19.718993 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.719709 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.721067 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.721303 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.721381 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.724361 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:19.724363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:19.724363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:19.724375 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:19.725987 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.726517 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.726518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.726519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.726586 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.726828 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.726829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.726829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.727025 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:19.727088 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.727422 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.727524 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:52:19.727883 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.715890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:51219)
0619 02:52:19.716080 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.716131 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.716774 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.717293 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.717300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.718450 (+ 1150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:19.718823 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.718837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.719855 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.719857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.720911 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.720916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.721419 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.721420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.721596 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.722297 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:19.722308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:19.724211 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:19.726119 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.726133 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.726136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.726436 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.726647 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:19.726647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:19.726648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:19.726768 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:19.727084 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:19.727092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:19.727269 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.727393 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.727605 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260619 02:52:19.728735 8747 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:19.728873 8747 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:19.729087 8747 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:19.729642 8288 ts_manager.cc:194] Registered new tserver with Master: 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
I20260619 02:52:19.730121 8288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:51219
I20260619 02:52:19.730687 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:19.739732 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.732414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.26.98.62:35015)
0619 02:52:19.732495 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.732498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.732653 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:52:19.732700 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.732701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.732711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.732767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:19.733473 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.734024 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.734984 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.735200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.735263 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.736739 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:19.736740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:19.736741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:19.736749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:19.738204 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.738619 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.738619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.738620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.738682 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:19.738966 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:19.738967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:19.738967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:19.739175 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:19.739271 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.739485 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.739570 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:52:19.742635 8288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:19.745400 8288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:19.754534 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.750128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.26.98.3:41135)
0619 02:52:19.750341 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.750344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.750640 (+ 296us) server_negotiation.cc:407] Connection header received
0619 02:52:19.750692 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.750694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.750711 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.750787 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:19.750950 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.752123 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.753517 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754073 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754145 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754222 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.754369 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.754425 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260619 02:52:19.754549 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.749809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.0.0.1:36880)
0619 02:52:19.750517 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.750525 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.750572 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.750912 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.750913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.750971 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:19.751036 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.751039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.752335 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.752336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.753258 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.753261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754288 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.754354 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754359 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.754410 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.754455 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":28,"thread_start_us":123,"threads_started":1}
I20260619 02:52:19.754643 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.750042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.26.98.2:35043)
0619 02:52:19.750266 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.750272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.750284 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:19.750338 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.750340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.750358 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.750455 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:19.750796 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.751757 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.753293 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754101 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754172 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754242 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.754504 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.754557 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260619 02:52:19.754866 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.749808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:35043 (local address 127.0.0.1:55544)
0619 02:52:19.750027 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.750072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.750188 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.750581 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.750584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.750587 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:19.750693 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.750696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.751866 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.751867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.753191 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.753195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754282 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.754370 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754373 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.754415 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.754456 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":7,"thread_start_us":110,"threads_started":1}
I20260619 02:52:19.755151 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.750188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.26.98.1:42079)
0619 02:52:19.750514 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.750518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.750568 (+ 50us) server_negotiation.cc:407] Connection header received
0619 02:52:19.750704 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.750707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.750724 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.750800 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:19.751146 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.752243 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.753359 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754101 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754172 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754259 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.754989 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.755057 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
I20260619 02:52:19.755225 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.749808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:41135 (local address 127.0.0.1:45712)
0619 02:52:19.750380 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.750389 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.750447 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.750806 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.750807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.750810 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:19.750858 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.750859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.752236 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.752237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.753440 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.753442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.754162 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.754163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.754257 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.754273 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.754345 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.754418 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":58,"thread_start_us":84,"threads_started":1}
I20260619 02:52:19.755839 8547 tablet_service.cc:1467] Processing CreateTablet for tablet 9293f25d299948a2ad32e49922286cdd (DEFAULT_TABLE table=test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:19.755878 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 9293f25d299948a2ad32e49922286cdd (DEFAULT_TABLE table=test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:19.755956 8681 tablet_service.cc:1467] Processing CreateTablet for tablet 9293f25d299948a2ad32e49922286cdd (DEFAULT_TABLE table=test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:19.756289 8547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9293f25d299948a2ad32e49922286cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:19.756320 8681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9293f25d299948a2ad32e49922286cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:19.756320 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9293f25d299948a2ad32e49922286cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:19.760043 8764 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Bootstrap starting.
I20260619 02:52:19.760634 8766 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap starting.
I20260619 02:52:19.760772 8765 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Bootstrap starting.
I20260619 02:52:19.761018 8764 tablet_bootstrap.cc:654] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:19.761410 8766 tablet_bootstrap.cc:654] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:19.761425 8765 tablet_bootstrap.cc:654] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:19.761564 8764 log.cc:824] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:19.762043 8765 log.cc:824] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:19.762045 8766 log.cc:824] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:19.762437 8764 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: No bootstrap required, opened a new log
I20260619 02:52:19.762526 8764 ts_tablet_manager.cc:1397] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:19.762823 8766 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: No bootstrap required, opened a new log
I20260619 02:52:19.762928 8766 ts_tablet_manager.cc:1397] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:19.762992 8765 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: No bootstrap required, opened a new log
I20260619 02:52:19.763082 8765 ts_tablet_manager.cc:1397] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:19.764508 8764 raft_consensus.cc:348] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.764684 8764 raft_consensus.cc:374] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:19.764717 8764 raft_consensus.cc:729] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4749dc9ee18b4dafae4195f902057137, State: Initialized, Role: FOLLOWER
I20260619 02:52:19.764876 8764 consensus_queue.cc:260] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.765200 8613 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
I20260619 02:52:19.765221 8764 ts_tablet_manager.cc:1428] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:19.765123 8766 raft_consensus.cc:348] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.765283 8766 raft_consensus.cc:374] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:19.765316 8766 raft_consensus.cc:729] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80def822f9d42a2ba2a81b5d0a1a33a, State: Initialized, Role: FOLLOWER
I20260619 02:52:19.765244 8765 raft_consensus.cc:348] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.765383 8765 raft_consensus.cc:374] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:19.765411 8765 raft_consensus.cc:729] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad7dec64be141888303b1f697a54de6, State: Initialized, Role: FOLLOWER
I20260619 02:52:19.765434 8766 consensus_queue.cc:260] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.765555 8765 consensus_queue.cc:260] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.765753 8766 ts_tablet_manager.cc:1428] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:19.765839 8765 ts_tablet_manager.cc:1428] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:19.765872 8479 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
I20260619 02:52:19.765877 8747 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
W20260619 02:52:19.783619 8614 tablet.cc:2369] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:19.825320 8480 tablet.cc:2369] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:19.955673 8772 raft_consensus.cc:482] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:19.955910 8772 raft_consensus.cc:504] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.956478 8772 leader_election.cc:288] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
I20260619 02:52:19.959689 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.956736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.26.98.3:41267)
0619 02:52:19.956799 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.956813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.956875 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.957086 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.957087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.957089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:19.957179 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.957182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.957925 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.957926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.958762 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.958765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.959383 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.959385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.959477 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.959480 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.959529 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.959585 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 02:52:19.959689 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.956890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:41267 (local address 127.26.98.1:42079)
0619 02:52:19.956963 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.956965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.956973 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:19.957003 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.957004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.957017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.957073 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:19.957241 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.957838 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.958860 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.959302 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.959367 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.959382 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.959543 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.959591 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:52:19.959898 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.956837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:45627 (local address 127.26.98.2:35043)
0619 02:52:19.956903 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.956905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.957232 (+ 327us) server_negotiation.cc:407] Connection header received
0619 02:52:19.957299 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.957300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.957336 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.957414 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:19.957577 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.958089 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.959072 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.959509 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.959573 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.959586 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.959754 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.959811 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:52:19.960161 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.956740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:35043 (local address 127.26.98.3:45627)
0619 02:52:19.957130 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.957141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:19.957238 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:19.957424 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:19.957424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:19.957426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:19.957511 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.957513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.958171 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.958171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.958995 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:19.958998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.959582 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.959583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:19.959661 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.959665 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:19.960031 (+ 366us) client_negotiation.cc:240] Negotiation successful
0619 02:52:19.960084 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":184,"thread_start_us":139,"threads_started":1}
I20260619 02:52:19.960156 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9293f25d299948a2ad32e49922286cdd" candidate_uuid: "5ad7dec64be141888303b1f697a54de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" is_pre_election: true
I20260619 02:52:19.960377 8433 raft_consensus.cc:2454] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad7dec64be141888303b1f697a54de6 in term 0.
I20260619 02:52:19.960547 8566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9293f25d299948a2ad32e49922286cdd" candidate_uuid: "5ad7dec64be141888303b1f697a54de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4749dc9ee18b4dafae4195f902057137" is_pre_election: true
I20260619 02:52:19.960695 8635 leader_election.cc:302] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad7dec64be141888303b1f697a54de6, d80def822f9d42a2ba2a81b5d0a1a33a; no voters:
I20260619 02:52:19.960752 8566 raft_consensus.cc:2454] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad7dec64be141888303b1f697a54de6 in term 0.
I20260619 02:52:19.960851 8772 raft_consensus.cc:2781] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:19.960937 8772 raft_consensus.cc:482] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:19.960973 8772 raft_consensus.cc:3037] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:19.961820 8772 raft_consensus.cc:504] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.961977 8772 leader_election.cc:288] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [CANDIDATE]: Term 1 election: Requested vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
I20260619 02:52:19.962209 8566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9293f25d299948a2ad32e49922286cdd" candidate_uuid: "5ad7dec64be141888303b1f697a54de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4749dc9ee18b4dafae4195f902057137"
I20260619 02:52:19.962205 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9293f25d299948a2ad32e49922286cdd" candidate_uuid: "5ad7dec64be141888303b1f697a54de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
I20260619 02:52:19.962292 8566 raft_consensus.cc:3037] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:19.962294 8433 raft_consensus.cc:3037] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:19.963207 8566 raft_consensus.cc:2454] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad7dec64be141888303b1f697a54de6 in term 1.
I20260619 02:52:19.963250 8433 raft_consensus.cc:2454] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad7dec64be141888303b1f697a54de6 in term 1.
I20260619 02:52:19.963413 8637 leader_election.cc:302] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6; no voters:
I20260619 02:52:19.963534 8772 raft_consensus.cc:2781] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:19.963729 8772 raft_consensus.cc:686] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad7dec64be141888303b1f697a54de6, State: Running, Role: LEADER
I20260619 02:52:19.963840 8772 consensus_queue.cc:237] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:19.965116 8288 catalog_manager.cc:5495] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad7dec64be141888303b1f697a54de6 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "5ad7dec64be141888303b1f697a54de6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } health_report { overall_health: HEALTHY } } }
W20260619 02:52:19.965682 8748 tablet.cc:2369] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:19.987782 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.985089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.26.98.3:41135)
0619 02:52:19.985178 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:19.985180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:19.985188 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:19.985234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:19.985236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:19.985262 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:19.985331 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:19.985482 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.986105 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.986949 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:19.987157 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:19.987216 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:19.987371 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:19.987518 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:19.987603 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:19.987622 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:52:19.987648 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:52:19.990523 8566 raft_consensus.cc:1261] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Refusing update from remote peer 5ad7dec64be141888303b1f697a54de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:19.990523 8433 raft_consensus.cc:1261] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Refusing update from remote peer 5ad7dec64be141888303b1f697a54de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:19.991015 8772 consensus_queue.cc:1035] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:19.991194 8774 consensus_queue.cc:1035] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:19.995716 8778 mvcc.cc:205] Tried to move back new op lower bound from 7298406563796246528 to 7298406563692625920. Current Snapshot: MvccSnapshot[applied={T|T < 7298406563796246528}]
I20260619 02:52:19.996899 8288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:19.997017 8288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:19.998152 8288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Sending DeleteTablet for 3 replicas of tablet 9293f25d299948a2ad32e49922286cdd
I20260619 02:52:19.998629 8681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9293f25d299948a2ad32e49922286cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:19 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45712
I20260619 02:52:19.998652 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 9293f25d299948a2ad32e49922286cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:19 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:36880
I20260619 02:52:19.998641 8547 tablet_service.cc:1514] Processing DeleteTablet for tablet 9293f25d299948a2ad32e49922286cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:19 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:55544
I20260619 02:52:19.999042 8789 tablet_replica.cc:330] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: stopping tablet replica
I20260619 02:52:19.999051 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8350
I20260619 02:52:19.999209 8789 raft_consensus.cc:2229] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:19.999363 8789 raft_consensus.cc:2258] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:20.000272 8789 ts_tablet_manager.cc:1905] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:20.003116 8788 tablet_replica.cc:330] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: stopping tablet replica
I20260619 02:52:20.003352 8788 raft_consensus.cc:2229] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:52:20.003562 8788 raft_consensus.cc:2258] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:20.004190 8788 ts_tablet_manager.cc:1905] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:20.004501 8789 ts_tablet_manager.cc:1918] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:20.004681 8789 log.cc:1196] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/9293f25d299948a2ad32e49922286cdd
I20260619 02:52:20.005102 8789 ts_tablet_manager.cc:1939] T 9293f25d299948a2ad32e49922286cdd P 4749dc9ee18b4dafae4195f902057137: Deleting consensus metadata
I20260619 02:52:20.005908 8275 catalog_manager.cc:4841] TS 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): tablet 9293f25d299948a2ad32e49922286cdd (table test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]) successfully deleted
I20260619 02:52:20.006122 8788 ts_tablet_manager.cc:1918] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:20.006219 8788 log.cc:1196] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/9293f25d299948a2ad32e49922286cdd
I20260619 02:52:20.006485 8788 ts_tablet_manager.cc:1939] T 9293f25d299948a2ad32e49922286cdd P 5ad7dec64be141888303b1f697a54de6: Deleting consensus metadata
I20260619 02:52:20.007069 8272 catalog_manager.cc:4841] TS 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): tablet 9293f25d299948a2ad32e49922286cdd (table test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]) successfully deleted
W20260619 02:52:20.007361 8273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:42079 (error 108)
I20260619 02:52:20.008083 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8484
I20260619 02:52:20.008347 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.008007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.0.0.1:36892)
0619 02:52:20.008115 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.008295 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 02:52:20.010648 8273 catalog_manager.cc:4584] TS d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9293f25d299948a2ad32e49922286cdd: Network error: Client connection negotiation failed: client connection to 127.26.98.1:42079: connect: Connection refused (error 111)
I20260619 02:52:20.016532 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8618
I20260619 02:52:20.022996 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8256
I20260619 02:52:20.030259 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--webserver_interface=127.26.98.62
--webserver_port=43259
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:20.134042 8791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:20.134696 8791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:20.134924 8791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:20.137769 8791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:20.137882 8791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:20.137900 8791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:20.137912 8791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:20.137924 8791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:20.141229 8791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=43259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:20.141716 8791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:20.142053 8791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:20.154212 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.154289 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.154372 8791 server_base.cc:1034] running on GCE node
W20260619 02:52:20.154280 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.155233 8791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:20.155746 8791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:20.156951 8791 hybrid_clock.cc:648] HybridClock initialized: now 1781837540156906 us; error 80 us; skew 500 ppm
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:20.160262 8791 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:20.160915 8791 webserver.cc:460] Webserver started at http://127.26.98.62:43259/ using document root <none> and password file <none>
I20260619 02:52:20.161182 8791 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:20.161257 8791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:20.162930 8791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:20.163827 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.164273 8791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:20.164390 8791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0a29b8a3f271444abd131d88328d1d0f"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "b064016d3c1d4d79e40b4eef5d4faf02"
server_key_iv: "4ac99d0ad4b79216d9ea1c729b979c69"
server_key_version: "encryptionkey@0"
I20260619 02:52:20.164963 8791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:20.179054 8791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:20.183377 8791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:20.183557 8791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:20.189079 8791 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35015
I20260619 02:52:20.189172 8859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35015 every 8 connection(s)
I20260619 02:52:20.189656 8791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:20.192339 8860 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:20.193432 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap starting.
I20260619 02:52:20.198336 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8791
I20260619 02:52:20.198704 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:42079
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=33903
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:20.205026 8860 log.cc:824] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:20.207865 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:20.208312 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap complete.
I20260619 02:52:20.211977 8860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:20.212396 8860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Initialized, Role: FOLLOWER
I20260619 02:52:20.212607 8860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:20.212703 8860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:20.212772 8860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:20.212821 8860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:52:20.214671 8860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:20.214828 8860 leader_election.cc:302] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a29b8a3f271444abd131d88328d1d0f; no voters:
I20260619 02:52:20.215118 8860 leader_election.cc:288] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:52:20.215186 8864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:52:20.215306 8864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Running, Role: LEADER
I20260619 02:52:20.215425 8864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:20.215468 8860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:20.215744 8865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:20.215827 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a29b8a3f271444abd131d88328d1d0f. Latest consensus state: current_term: 2 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:20.215914 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:20.215839 8865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:20.216156 8868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:20.217258 8868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:20.217391 8868 catalog_manager.cc:1259] Loaded cluster ID: 2f7d30b8814e473da31621672f3b2692
I20260619 02:52:20.217657 8868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:20.217921 8868 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:52:20.220686 8868 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:52:20.224826 8868 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:20.225175 8868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Loaded TSK: 0
I20260619 02:52:20.225412 8868 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:20.316164 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:20.316516 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:20.316565 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:20.318240 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:20.318317 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:20.318367 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:20.320408 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:42079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=33903
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:20.320761 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:20.321038 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:20.328665 8888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.328670 8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.328665 8887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.329676 8862 server_base.cc:1034] running on GCE node
I20260619 02:52:20.330056 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:20.330461 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:20.331730 8862 hybrid_clock.cc:648] HybridClock initialized: now 1781837540331693 us; error 101 us; skew 500 ppm
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:20.336036 8862 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:20.336756 8862 webserver.cc:460] Webserver started at http://127.26.98.1:33903/ using document root <none> and password file <none>
I20260619 02:52:20.337055 8862 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:20.337172 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:20.339485 8862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:20.340327 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.340586 8862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:20.340734 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "93e34039cacae4cae02c624001ab6d5d"
server_key_iv: "70dd7ba2696b67069290c88f20261ad6"
server_key_version: "encryptionkey@0"
I20260619 02:52:20.341459 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:20.357295 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:20.360852 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:20.361047 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:20.361378 8862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:20.362142 8904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:20.364445 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:20.364543 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260619 02:52:20.364586 8862 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:20.365746 8862 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:20.365820 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:20.365975 8904 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap starting.
I20260619 02:52:20.375252 8862 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:42079
I20260619 02:52:20.375818 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:20.376921 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 8862
I20260619 02:52:20.377476 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:35043
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=38629
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:20.380323 8904 log.cc:824] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:20.384541 9011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:42079 every 8 connection(s)
I20260619 02:52:20.384938 8904 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:20.385421 8904 tablet_bootstrap.cc:492] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap complete.
I20260619 02:52:20.386854 8904 ts_tablet_manager.cc:1397] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20260619 02:52:20.388823 8904 raft_consensus.cc:348] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:20.389269 8904 raft_consensus.cc:729] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80def822f9d42a2ba2a81b5d0a1a33a, State: Initialized, Role: FOLLOWER
I20260619 02:52:20.389470 8904 consensus_queue.cc:260] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } }
I20260619 02:52:20.389870 8904 ts_tablet_manager.cc:1428] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:20.398604 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.379062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:40481 (local address 127.26.98.62:35015)
0619 02:52:20.383039 (+ 3977us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.383052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.383120 (+ 68us) server_negotiation.cc:407] Connection header received
0619 02:52:20.384016 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.384023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.384113 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.384305 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:20.385924 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.387525 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.389500 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.390203 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.390345 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.393627 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:20.393633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:20.393633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:20.393651 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:20.395998 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.396766 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.396772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.396772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.396950 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.397282 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.397283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.397283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.397681 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:20.397792 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.398143 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.398386 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":114,"threads_started":1}
I20260619 02:52:20.398609 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.377081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:40481)
0619 02:52:20.381958 (+ 4877us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.382026 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.383582 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.384795 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.384802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.385287 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:20.385712 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.385723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.387921 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.387923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.389324 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.389328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.390320 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.390321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.390508 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.391284 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:20.391296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:20.393471 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:20.396210 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.396227 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.396230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.396646 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.397029 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.397029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.397030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.397191 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.397763 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:20.397769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:20.397956 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.398076 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.398313 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4809,"thread_start_us":111,"threads_started":1}
I20260619 02:52:20.399763 9012 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:20.399907 9012 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:20.400138 9012 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:20.401087 8824 ts_manager.cc:194] Registered new tserver with Master: d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
I20260619 02:52:20.402350 8824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:40481
I20260619 02:52:20.403831 9012 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
I20260619 02:52:20.413313 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.401878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.0.0.1:36902)
0619 02:52:20.402636 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.402709 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.402852 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.403441 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.403445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.403913 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:20.404002 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.404006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.405422 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.405424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.406205 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.406209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.406950 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.406951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.407044 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.407687 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:20.407692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:20.409508 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:20.411897 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.411909 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.411911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.412214 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.412477 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.412478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.412478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.412547 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.412887 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:20.412894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:20.412960 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.413035 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.413219 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":143,"threads_started":1}
I20260619 02:52:20.413347 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.402102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.26.98.1:42079)
0619 02:52:20.402392 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.402399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.402839 (+ 440us) server_negotiation.cc:407] Connection header received
0619 02:52:20.403214 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.403230 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.403325 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.403424 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:20.404132 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.405308 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.406306 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.406818 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.406939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.409642 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:20.409659 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:20.409659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:20.409671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:20.411753 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.412307 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.412311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.412311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.412408 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.412604 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.412605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.412605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.412821 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:20.412912 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.413090 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.413214 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":152,"threads_started":1}
I20260619 02:52:20.414144 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet 9293f25d299948a2ad32e49922286cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:19 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:36902
I20260619 02:52:20.415019 9021 tablet_replica.cc:330] stopping tablet replica
I20260619 02:52:20.415144 9021 raft_consensus.cc:2229] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:20.415230 9021 raft_consensus.cc:2258] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:20.415618 9021 ts_tablet_manager.cc:1905] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:20.418011 9021 ts_tablet_manager.cc:1918] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:20.418100 9021 log.cc:1196] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/9293f25d299948a2ad32e49922286cdd
I20260619 02:52:20.418335 9021 ts_tablet_manager.cc:1939] T 9293f25d299948a2ad32e49922286cdd P d80def822f9d42a2ba2a81b5d0a1a33a: Deleting consensus metadata
I20260619 02:52:20.419057 8809 catalog_manager.cc:4841] TS d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079): tablet 9293f25d299948a2ad32e49922286cdd (table test-table [id=8c0c89061f514e4fa0df00f8d9f11dbf]) successfully deleted
W20260619 02:52:20.489092 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:20.489418 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:20.489462 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:20.491297 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:20.491441 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:20.491497 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:20.493855 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:35043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=38629
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:20.494240 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:20.494534 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:20.501230 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.501451 9016 server_base.cc:1034] running on GCE node
W20260619 02:52:20.501242 9027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.501219 9029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.501785 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:20.502115 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:20.503335 9016 hybrid_clock.cc:648] HybridClock initialized: now 1781837540503312 us; error 33 us; skew 500 ppm
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:20.507526 9016 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:20.508219 9016 webserver.cc:460] Webserver started at http://127.26.98.2:38629/ using document root <none> and password file <none>
I20260619 02:52:20.508517 9016 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:20.508602 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:20.510871 9016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:20.511878 9036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.512248 9016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:20.512398 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "4749dc9ee18b4dafae4195f902057137"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "5c08a54f3f41008812fa8bc1968811cb"
server_key_iv: "1452d3c3589741ea15c3c9138915fb67"
server_key_version: "encryptionkey@0"
I20260619 02:52:20.512910 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:20.525952 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:20.529659 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:20.529886 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:20.530184 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:20.530565 9016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:20.530776 9016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.530846 9016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:20.530869 9016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.539006 9016 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:35043
I20260619 02:52:20.539058 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:35043 every 8 connection(s)
I20260619 02:52:20.539521 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:20.547504 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9016
I20260619 02:52:20.547768 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:41135
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=43761
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:20.552294 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.540474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:44833 (local address 127.26.98.62:35015)
0619 02:52:20.540545 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.540548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.540788 (+ 240us) server_negotiation.cc:407] Connection header received
0619 02:52:20.541441 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.541443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.541464 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.541540 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:20.542744 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.543375 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.544556 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.544792 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.544861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.548079 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:20.548082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:20.548082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:20.548094 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:20.549911 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.550653 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.550655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.550655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.550786 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.551257 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.551261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.551261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.551530 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:20.551620 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.552076 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.552205 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:52:20.552577 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.540414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:44833)
0619 02:52:20.540626 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.540671 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.541302 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.541638 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.541643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.542354 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:20.542612 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.542618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.543482 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.543483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.544441 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.544443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.544878 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.544879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.545040 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.545771 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:20.545782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:20.547914 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:20.550032 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.550049 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.550055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.550505 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.550923 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.550924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.550924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.551108 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.551633 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:20.551640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:20.551882 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.552022 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.552260 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260619 02:52:20.553373 9150 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:20.553617 9150 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:20.553929 9150 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:20.554680 8824 ts_manager.cc:194] Registered new tserver with Master: 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043)
I20260619 02:52:20.555191 8824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:44833
W20260619 02:52:20.651027 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:20.651367 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:20.651417 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:20.653160 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:20.653258 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:20.653311 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:20.656528 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:41135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=43761
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:20.657137 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:20.657482 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:20.663949 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.663825 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.663823 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.664433 9153 server_base.cc:1034] running on GCE node
I20260619 02:52:20.664682 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:20.664971 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:20.666184 9153 hybrid_clock.cc:648] HybridClock initialized: now 1781837540666170 us; error 40 us; skew 500 ppm
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:20.669899 9153 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:20.670598 9153 webserver.cc:460] Webserver started at http://127.26.98.3:43761/ using document root <none> and password file <none>
I20260619 02:52:20.670967 9153 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:20.671064 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:20.672945 9153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:52:20.674057 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.674506 9153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:20.674654 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "5ad7dec64be141888303b1f697a54de6"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "82ae602b93437208028e1fb91dc4ab7c"
server_key_iv: "6137f995e0e27284a3b014d1a7d89675"
server_key_version: "encryptionkey@0"
I20260619 02:52:20.675239 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:20.688457 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:20.691823 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:20.692020 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:20.692314 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:20.693029 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:20.693085 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.693128 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:20.693153 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.702014 9153 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:41135
I20260619 02:52:20.702100 9281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:41135 every 8 connection(s)
I20260619 02:52:20.702589 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:20.705819 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9153
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:20.715207 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.706352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.26.98.62:35015)
0619 02:52:20.706481 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.706482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.706602 (+ 120us) server_negotiation.cc:407] Connection header received
0619 02:52:20.706636 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.706637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.706651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.706746 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:20.707517 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.708293 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.709257 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.709518 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.709573 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.710952 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:20.710954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:20.710954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:20.710964 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:20.712957 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.713665 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.713667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.713668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.713769 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.714139 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.714141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.714141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.714342 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:20.714531 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.714795 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.714974 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260619 02:52:20.716951 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.703756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:45367 (local address 127.26.98.62:35015)
0619 02:52:20.703847 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.703850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.704070 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:52:20.704790 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.704793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.704805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.704874 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:20.706117 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.706833 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.708505 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.708739 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.708803 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.712368 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:20.712370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:20.712370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:20.712381 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:20.714599 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.715506 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.715507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.715508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.715686 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:20.716024 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:20.716025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:20.716025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:20.716240 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:20.716308 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.716737 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.716846 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:52:20.717243 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.703719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:45367)
0619 02:52:20.703906 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.703957 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.704644 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.704951 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.704955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.705647 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:20.705945 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.705963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.707010 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.707011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.708420 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.708424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.708845 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.708846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.709037 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.709814 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:20.709826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:20.712196 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:20.714768 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.714790 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.714794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.715422 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.715764 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:20.715765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:20.715765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:20.715951 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:20.716331 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:20.716342 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:20.716578 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.716682 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.716972 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260619 02:52:20.717878 9282 heartbeater.cc:347] Connected to a master server at 127.26.98.62:35015
I20260619 02:52:20.718004 9282 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:20.718235 9282 heartbeater.cc:511] Master 127.26.98.62:35015 requested a full tablet report, sending...
I20260619 02:52:20.718950 8824 ts_manager.cc:194] Registered new tserver with Master: 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
I20260619 02:52:20.719385 8824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:45367
I20260619 02:52:20.728056 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:20.732141 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.728584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.26.98.62:35015)
0619 02:52:20.728733 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.728737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.728963 (+ 226us) server_negotiation.cc:407] Connection header received
0619 02:52:20.729058 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.729059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.729114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.729189 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:20.729372 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.729949 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.730963 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.731192 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.731248 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.731604 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:20.731760 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:20.731881 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.731986 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.732028 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 02:52:20.732771 8824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:20.737092 8824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:20.741519 8946 tablet_service.cc:1467] Processing CreateTablet for tablet 99d652c42297464c96e49af262801da5 (DEFAULT_TABLE table=test-table [id=15057f336e1f4b0fb9e65dac39b57885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:20.741889 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d652c42297464c96e49af262801da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:20.744583 8904 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap starting.
I20260619 02:52:20.745139 8904 tablet_bootstrap.cc:654] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:20.746229 8904 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: No bootstrap required, opened a new log
I20260619 02:52:20.746308 8904 ts_tablet_manager.cc:1397] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:52:20.746522 8904 raft_consensus.cc:348] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.746687 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.741288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:41135 (local address 127.0.0.1:45722)
0619 02:52:20.741367 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.741381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.741430 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.742667 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.742669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.742673 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:20.742753 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.742756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.744539 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.744541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.745406 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.745410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.746359 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.746360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.746451 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.746493 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.746526 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.746598 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:52:20.746647 8904 raft_consensus.cc:374] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:20.746729 8904 raft_consensus.cc:729] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80def822f9d42a2ba2a81b5d0a1a33a, State: Initialized, Role: FOLLOWER
I20260619 02:52:20.746786 8904 consensus_queue.cc:260] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.746932 8904 ts_tablet_manager.cc:1428] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:20.747141 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.741607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.26.98.3:41135)
0619 02:52:20.742204 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.742211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.742226 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:52:20.742279 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.742282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.742303 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.742860 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:20.743055 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.744272 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.745582 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.746252 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.746805 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.746942 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.746977 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.747039 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":126,"threads_started":1}
I20260619 02:52:20.747422 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.741884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.26.98.2:35043)
0619 02:52:20.742279 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.742286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.742300 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:52:20.742351 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.742354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.742372 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.742464 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:20.742722 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.743590 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.746156 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.746864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.746961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.747032 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.747240 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.747320 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":93,"threads_started":1}
I20260619 02:52:20.747745 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.741288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:35043 (local address 127.0.0.1:55556)
0619 02:52:20.741508 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.741517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.741567 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.742583 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.742584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.742586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:20.742659 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.742662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.743711 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.743712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.746045 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.746049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.747071 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.747072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.747146 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.747152 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.747515 (+ 363us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.747665 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":41,"thread_start_us":115,"threads_started":1}
I20260619 02:52:20.748442 9216 tablet_service.cc:1467] Processing CreateTablet for tablet 99d652c42297464c96e49af262801da5 (DEFAULT_TABLE table=test-table [id=15057f336e1f4b0fb9e65dac39b57885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:20.748890 9216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d652c42297464c96e49af262801da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:20.753187 9295 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Bootstrap starting.
I20260619 02:52:20.754257 9295 tablet_bootstrap.cc:654] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:20.754102 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 99d652c42297464c96e49af262801da5 (DEFAULT_TABLE table=test-table [id=15057f336e1f4b0fb9e65dac39b57885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:20.754580 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d652c42297464c96e49af262801da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:20.754967 9295 log.cc:824] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:20.757158 9295 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: No bootstrap required, opened a new log
I20260619 02:52:20.757318 9295 ts_tablet_manager.cc:1397] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:52:20.759871 9295 raft_consensus.cc:348] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.760129 9295 raft_consensus.cc:374] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:20.760167 9295 raft_consensus.cc:729] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad7dec64be141888303b1f697a54de6, State: Initialized, Role: FOLLOWER
I20260619 02:52:20.760356 9295 consensus_queue.cc:260] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.760553 9297 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Bootstrap starting.
I20260619 02:52:20.760727 9295 ts_tablet_manager.cc:1428] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:52:20.760803 9282 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
I20260619 02:52:20.761503 9297 tablet_bootstrap.cc:654] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:20.762248 9297 log.cc:824] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:20.763500 9297 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: No bootstrap required, opened a new log
I20260619 02:52:20.763602 9297 ts_tablet_manager.cc:1397] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:20.765367 9297 raft_consensus.cc:348] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.765654 9297 raft_consensus.cc:374] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:20.765708 9297 raft_consensus.cc:729] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4749dc9ee18b4dafae4195f902057137, State: Initialized, Role: FOLLOWER
I20260619 02:52:20.765889 9297 consensus_queue.cc:260] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.766291 9297 ts_tablet_manager.cc:1428] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:20.766347 9150 heartbeater.cc:503] Master 127.26.98.62:35015 was elected leader, sending a full tablet report...
I20260619 02:52:20.785398 9300 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:20.785584 9300 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.786124 9300 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135), d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
W20260619 02:52:20.790298 9151 tablet.cc:2369] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:20.791323 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.786474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:45047 (local address 127.26.98.3:41135)
0619 02:52:20.786541 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.786543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.786793 (+ 250us) server_negotiation.cc:407] Connection header received
0619 02:52:20.786824 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.786825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.786841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.786934 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:20.787279 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.788133 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.790229 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.790832 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.790982 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.791000 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.791180 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.791239 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:52:20.791371 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.786448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:41135 (local address 127.26.98.2:45047)
0619 02:52:20.786695 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.786704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.786775 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.787011 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.787013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.787016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:20.787164 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.787167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.788264 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.788265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.790017 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.790024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.791005 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.791007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.791094 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.791099 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.791165 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.791231 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":137,"thread_start_us":162,"threads_started":1}
I20260619 02:52:20.791563 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.786554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:56253 (local address 127.26.98.1:42079)
0619 02:52:20.786614 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.786616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.786629 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:52:20.786726 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.786727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.786753 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.786901 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:20.787283 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.788149 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.790219 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.790867 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.790973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.791065 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.791412 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.791487 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:52:20.791378 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.786428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.26.98.2:56253)
0619 02:52:20.786564 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.786581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:20.786643 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:20.787011 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:20.787013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:20.787016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:20.787164 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.787167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.788264 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.788265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.790018 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:20.790024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.791057 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.791059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:20.791147 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.791151 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:20.791214 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:52:20.791281 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":326}
I20260619 02:52:20.791953 9236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d652c42297464c96e49af262801da5" candidate_uuid: "4749dc9ee18b4dafae4195f902057137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad7dec64be141888303b1f697a54de6" is_pre_election: true
I20260619 02:52:20.792172 9236 raft_consensus.cc:2454] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4749dc9ee18b4dafae4195f902057137 in term 0.
I20260619 02:52:20.792096 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d652c42297464c96e49af262801da5" candidate_uuid: "4749dc9ee18b4dafae4195f902057137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" is_pre_election: true
I20260619 02:52:20.792291 8954 raft_consensus.cc:2454] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4749dc9ee18b4dafae4195f902057137 in term 0.
I20260619 02:52:20.792483 9037 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6; no voters:
I20260619 02:52:20.792742 9300 raft_consensus.cc:2781] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:20.792816 9300 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:20.792842 9300 raft_consensus.cc:3037] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:20.793843 9300 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.794036 9300 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [CANDIDATE]: Term 1 election: Requested vote from peers 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135), d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079)
I20260619 02:52:20.794327 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d652c42297464c96e49af262801da5" candidate_uuid: "4749dc9ee18b4dafae4195f902057137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
I20260619 02:52:20.794432 8954 raft_consensus.cc:3037] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:20.795074 9236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d652c42297464c96e49af262801da5" candidate_uuid: "4749dc9ee18b4dafae4195f902057137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad7dec64be141888303b1f697a54de6"
I20260619 02:52:20.795320 9236 raft_consensus.cc:3037] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:20.795413 8954 raft_consensus.cc:2454] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4749dc9ee18b4dafae4195f902057137 in term 1.
I20260619 02:52:20.795640 9038 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4749dc9ee18b4dafae4195f902057137, d80def822f9d42a2ba2a81b5d0a1a33a; no voters:
I20260619 02:52:20.795744 9300 raft_consensus.cc:2781] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:20.795940 9300 raft_consensus.cc:686] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 1 LEADER]: Becoming Leader. State: Replica: 4749dc9ee18b4dafae4195f902057137, State: Running, Role: LEADER
I20260619 02:52:20.796072 9300 consensus_queue.cc:237] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:20.796448 9236 raft_consensus.cc:2454] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4749dc9ee18b4dafae4195f902057137 in term 1.
I20260619 02:52:20.797159 8824 catalog_manager.cc:5495] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4749dc9ee18b4dafae4195f902057137 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "4749dc9ee18b4dafae4195f902057137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260619 02:52:20.810252 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.807134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.26.98.2:35043)
0619 02:52:20.807228 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:20.807230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:20.807240 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:20.807304 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:20.807305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:20.807329 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:20.807390 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:20.807689 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.808234 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.809204 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:20.809440 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:20.809517 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:20.809790 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:20.809947 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:20.810027 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:20.810088 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:52:20.810115 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:52:20.813001 9236 raft_consensus.cc:1261] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Refusing update from remote peer 4749dc9ee18b4dafae4195f902057137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:20.813020 8954 raft_consensus.cc:1261] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Refusing update from remote peer 4749dc9ee18b4dafae4195f902057137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:20.813475 9300 consensus_queue.cc:1035] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:20.813673 9302 consensus_queue.cc:1035] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:20.817718 9307 mvcc.cc:205] Tried to move back new op lower bound from 7298406567166169088 to 7298406567101435904. Current Snapshot: MvccSnapshot[applied={T|T < 7298406567166169088 or (T in {7298406567166169088})}]
I20260619 02:52:20.821620 8823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:20.821767 8823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:20.822736 8823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Sending DeleteTablet for 3 replicas of tablet 99d652c42297464c96e49af262801da5
I20260619 02:52:20.823170 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d652c42297464c96e49af262801da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:55556
I20260619 02:52:20.823179 9216 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d652c42297464c96e49af262801da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:20 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45722
I20260619 02:52:20.823259 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8862
I20260619 02:52:20.824497 9316 tablet_replica.cc:330] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: stopping tablet replica
I20260619 02:52:20.824689 9316 raft_consensus.cc:2229] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:20.824847 9316 raft_consensus.cc:2258] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:20.825476 9316 ts_tablet_manager.cc:1905] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:20.827162 9315 tablet_replica.cc:330] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: stopping tablet replica
I20260619 02:52:20.827499 9315 raft_consensus.cc:2229] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:52:20.827798 9316 ts_tablet_manager.cc:1918] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:20.827811 9315 raft_consensus.cc:2258] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:20.827908 9316 log.cc:1196] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/99d652c42297464c96e49af262801da5
I20260619 02:52:20.828248 9316 ts_tablet_manager.cc:1939] T 99d652c42297464c96e49af262801da5 P 5ad7dec64be141888303b1f697a54de6: Deleting consensus metadata
I20260619 02:52:20.828886 8808 catalog_manager.cc:4841] TS 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): tablet 99d652c42297464c96e49af262801da5 (table test-table [id=15057f336e1f4b0fb9e65dac39b57885]) successfully deleted
I20260619 02:52:20.828925 9315 ts_tablet_manager.cc:1905] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:52:20.831243 8809 connection.cc:535] client connection to 127.26.98.1:42079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:52:20.831369 9315 ts_tablet_manager.cc:1918] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 02:52:20.831413 8809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:52:20.831506 9315 log.cc:1196] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/99d652c42297464c96e49af262801da5
I20260619 02:52:20.831902 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9016
I20260619 02:52:20.834183 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.833964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:42079 (local address 127.0.0.1:36904)
0619 02:52:20.834051 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.834121 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 02:52:20.834437 8809 catalog_manager.cc:4584] TS d80def822f9d42a2ba2a81b5d0a1a33a (127.26.98.1:42079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99d652c42297464c96e49af262801da5: Network error: Client connection negotiation failed: client connection to 127.26.98.1:42079: connect: Connection refused (error 111)
I20260619 02:52:20.838436 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9153
I20260619 02:52:20.838618 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.838461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:35043 (local address 127.0.0.1:55584)
0619 02:52:20.838531 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.838566 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:35043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 02:52:20.844193 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 8791
I20260619 02:52:20.850239 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--webserver_interface=127.26.98.62
--webserver_port=43259
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:20.956008 9318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:20.956359 9318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:20.956403 9318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:20.958048 9318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:20.958137 9318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:20.958159 9318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:20.958178 9318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:20.958194 9318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:20.960458 9318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=43259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:20.960891 9318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:20.961185 9318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:20.967440 9324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.967424 9323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:20.967424 9326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:20.967465 9318 server_base.cc:1034] running on GCE node
I20260619 02:52:20.968138 9318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:20.968518 9318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:20.969762 9318 hybrid_clock.cc:648] HybridClock initialized: now 1781837540969747 us; error 43 us; skew 500 ppm
Jun 19 02:52:20 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837540, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:20.973650 9318 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:20.974414 9318 webserver.cc:460] Webserver started at http://127.26.98.62:43259/ using document root <none> and password file <none>
I20260619 02:52:20.974750 9318 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:20.974818 9318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:20.976912 9318 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:20.978013 9333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:20.978349 9318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:20.978487 9318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0a29b8a3f271444abd131d88328d1d0f"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "b064016d3c1d4d79e40b4eef5d4faf02"
server_key_iv: "4ac99d0ad4b79216d9ea1c729b979c69"
server_key_version: "encryptionkey@0"
I20260619 02:52:20.979105 9318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:21.002465 9318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:21.005525 9318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:21.005724 9318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:21.011224 9318 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35015
I20260619 02:52:21.011291 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35015 every 8 connection(s)
I20260619 02:52:21.011799 9318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:21.013811 9386 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:21.014626 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap starting.
I20260619 02:52:21.018805 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9318
I20260619 02:52:21.019177 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:42079
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=33903
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:21.026501 9386 log.cc:824] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:21.029539 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:21.029945 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f: Bootstrap complete.
I20260619 02:52:21.032771 9386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:21.033092 9386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Initialized, Role: FOLLOWER
I20260619 02:52:21.033284 9386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:21.033368 9386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:21.033424 9386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:21.033476 9386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:52:21.034832 9386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:21.035010 9386 leader_election.cc:302] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a29b8a3f271444abd131d88328d1d0f; no voters:
I20260619 02:52:21.035230 9386 leader_election.cc:288] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:52:21.035297 9390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:52:21.035430 9390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b8a3f271444abd131d88328d1d0f, State: Running, Role: LEADER
I20260619 02:52:21.035539 9390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } }
I20260619 02:52:21.035614 9386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:21.035868 9390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a29b8a3f271444abd131d88328d1d0f. Latest consensus state: current_term: 3 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:21.035954 9390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:21.035876 9391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0a29b8a3f271444abd131d88328d1d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a29b8a3f271444abd131d88328d1d0f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35015 } } }
I20260619 02:52:21.036001 9391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a29b8a3f271444abd131d88328d1d0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:21.036360 9394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:21.037449 9394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:21.037634 9394 catalog_manager.cc:1259] Loaded cluster ID: 2f7d30b8814e473da31621672f3b2692
I20260619 02:52:21.037853 9394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:52:21.038177 9394 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c7f69d9ad google::LogMessage::Fail() at ??:0
@ 0x7f9c7f6a088a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c7f69d4de google::LogMessage::Flush() at ??:0
@ 0x7f9c7f69ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c875be0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9c875c78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c8156b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c81560954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c823c46db start_thread at ??:0
@ 0x7f9c7e19b61f clone at ??:0
W20260619 02:52:21.150003 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:21.150282 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:21.150313 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:21.151834 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:21.151906 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:21.151942 9388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:21.153839 9388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:42079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=33903
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:21.154177 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:21.154462 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:21.161149 9422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.161150 9423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.161150 9425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:21.161587 9388 server_base.cc:1034] running on GCE node
I20260619 02:52:21.161859 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:21.162195 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:21.163421 9388 hybrid_clock.cc:648] HybridClock initialized: now 1781837541163383 us; error 52 us; skew 500 ppm
Jun 19 02:52:21 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837541, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:21.167158 9388 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:21.167860 9388 webserver.cc:460] Webserver started at http://127.26.98.1:33903/ using document root <none> and password file <none>
I20260619 02:52:21.168133 9388 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:21.168200 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:21.170075 9388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:21.171000 9432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.171224 9388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:21.171341 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "d80def822f9d42a2ba2a81b5d0a1a33a"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "93e34039cacae4cae02c624001ab6d5d"
server_key_iv: "70dd7ba2696b67069290c88f20261ad6"
server_key_version: "encryptionkey@0"
I20260619 02:52:21.171875 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:21.192833 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:21.196324 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:21.196584 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:21.196929 9388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:21.197572 9439 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:21.199756 9388 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:21.199888 9388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:52:21.199937 9388 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:21.200989 9388 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:21.201049 9388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:21.201141 9439 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap starting.
I20260619 02:52:21.210630 9388 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:42079
I20260619 02:52:21.210690 9546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:42079 every 8 connection(s)
I20260619 02:52:21.211418 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:21.216365 9439 log.cc:824] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:21.216970 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9388
I20260619 02:52:21.217259 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:35043
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=38629
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:21.219950 9439 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:21.220291 9439 tablet_bootstrap.cc:492] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Bootstrap complete.
I20260619 02:52:21.221143 9439 ts_tablet_manager.cc:1397] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20260619 02:52:21.222462 9439 raft_consensus.cc:348] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:21.222771 9439 raft_consensus.cc:729] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80def822f9d42a2ba2a81b5d0a1a33a, State: Initialized, Role: FOLLOWER
I20260619 02:52:21.222973 9439 consensus_queue.cc:260] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:21.223357 9439 ts_tablet_manager.cc:1428] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260619 02:52:21.359828 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:21.360224 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:21.360277 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:21.362322 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:21.362406 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:21.362452 9551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:21.365063 9551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:35043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=38629
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:21.365442 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:21.365767 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:21.371742 9560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.371743 9558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.372098 9557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:21.372385 9551 server_base.cc:1034] running on GCE node
I20260619 02:52:21.372617 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:21.372907 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:21.374099 9551 hybrid_clock.cc:648] HybridClock initialized: now 1781837541374074 us; error 42 us; skew 500 ppm
Jun 19 02:52:21 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837541, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:21.378489 9551 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:21.379221 9551 webserver.cc:460] Webserver started at http://127.26.98.2:38629/ using document root <none> and password file <none>
I20260619 02:52:21.379645 9551 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:21.379720 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:21.381528 9551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:21.382393 9567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.382613 9551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:21.382725 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "4749dc9ee18b4dafae4195f902057137"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "5c08a54f3f41008812fa8bc1968811cb"
server_key_iv: "1452d3c3589741ea15c3c9138915fb67"
server_key_version: "encryptionkey@0"
I20260619 02:52:21.383374 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260619 02:52:21.385142 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:52:21.212486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:50027)
0619 02:52:21.212742 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.212810 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:21.213601 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:21.384914 (+171313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":108,"threads_started":1}
W20260619 02:52:21.385406 9547 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:35015 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:35015: Client connection negotiation failed: client connection to 127.26.98.62:35015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:52:21.385854 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.385672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:59309)
0619 02:52:21.385774 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.385823 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 02:52:21.386225 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.386136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:41759)
0619 02:52:21.386175 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.386201 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:52:21.386327 9547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:21.395032 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:21.398234 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:21.398614 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:21.399123 9551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:21.399758 9574 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 02:52:21.401597 9574 ts_tablet_manager.cc:1852] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 02:52:21.402026 9574 ts_tablet_manager.cc:1905] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:21.404987 9574 ts_tablet_manager.cc:1918] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:21.405206 9574 ts_tablet_manager.cc:1939] T 99d652c42297464c96e49af262801da5 P 4749dc9ee18b4dafae4195f902057137: Deleting consensus metadata
I20260619 02:52:21.405514 9551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 02:52:21.405593 9551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 02:52:21.405642 9551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:21.405660 9551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.413727 9551 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:35043
I20260619 02:52:21.413846 9681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:35043 every 8 connection(s)
I20260619 02:52:21.414239 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:21.415870 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.415183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:53859)
0619 02:52:21.415525 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.415669 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
W20260619 02:52:21.416270 9682 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:35015 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:35015: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
I20260619 02:52:21.416661 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.416543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:52305)
0619 02:52:21.416596 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.416633 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 02:52:21.416988 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.416914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:35491)
0619 02:52:21.416952 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.416973 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:52:21.417065 9682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:21.417300 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9551
I20260619 02:52:21.417574 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:41135
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:35015
--webserver_port=43761
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:21.519352 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:21.519701 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:21.519748 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:21.521348 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:21.521431 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:21.521482 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:21.523574 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:41135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=43761
--tserver_master_addrs=127.26.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:21.523958 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:21.524240 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:21.531064 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.531064 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:21.531064 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:21.531560 9685 server_base.cc:1034] running on GCE node
I20260619 02:52:21.531814 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:21.532117 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:21.533290 9685 hybrid_clock.cc:648] HybridClock initialized: now 1781837541533267 us; error 43 us; skew 500 ppm
Jun 19 02:52:21 dist-test-slave-0f9d krb5kdc[8243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837541, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:21.536827 9685 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:21.537490 9685 webserver.cc:460] Webserver started at http://127.26.98.3:43761/ using document root <none> and password file <none>
I20260619 02:52:21.537747 9685 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:21.537803 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:21.539659 9685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:52:21.540454 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.540645 9685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:21.540740 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "5ad7dec64be141888303b1f697a54de6"
format_stamp: "Formatted at 2026-06-19 02:52:19 on dist-test-slave-0f9d"
server_key: "82ae602b93437208028e1fb91dc4ab7c"
server_key_iv: "6137f995e0e27284a3b014d1a7d89675"
server_key_version: "encryptionkey@0"
I20260619 02:52:21.541186 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:21.552148 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:21.554790 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:21.555003 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:21.555284 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:21.555648 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:21.555685 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.555719 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:21.555743 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:21.564455 9685 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:41135
I20260619 02:52:21.564481 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:41135 every 8 connection(s)
I20260619 02:52:21.565006 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:21.566474 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.565924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:38169)
0619 02:52:21.566130 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.566231 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260619 02:52:21.567054 9814 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:35015 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:35015: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
I20260619 02:52:21.567422 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.567282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:34871)
0619 02:52:21.567359 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.567395 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:52:21.567960 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.567740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:34519)
0619 02:52:21.567900 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.567932 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260619 02:52:21.568107 9814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:21.575022 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9685
I20260619 02:52:22.349740 9818 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:22.349913 9818 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:22.350344 9818 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
Jun 19 02:52:22 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837541, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.2@KRBTEST.COM
Jun 19 02:52:22 dist-test-slave-0f9d krb5kdc[8243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837541, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.3@KRBTEST.COM
I20260619 02:52:22.361816 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.350591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:35043 (local address 127.26.98.1:54013)
0619 02:52:22.350727 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.350740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:22.350814 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:22.352095 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:22.352099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:22.352553 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:22.352784 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:22.352792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.353972 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.353973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:22.354812 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:22.354814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.355420 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.355421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:22.355537 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:22.356160 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:22.356164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:22.357810 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:22.360411 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:22.360412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:22.360541 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:22.360789 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:22.360790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:22.360848 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:22.361241 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:22.361250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:22.361379 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 02:52:22.361474 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:52:22.361706 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":96,"thread_start_us":58,"threads_started":1}
I20260619 02:52:22.361829 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.350810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:52739 (local address 127.26.98.3:41135)
0619 02:52:22.350993 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:22.351008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:22.351035 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:52:22.351688 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:22.351691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:22.351835 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:22.352008 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:22.352916 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.353864 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.354912 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.355341 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.355441 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:22.358262 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:22.358266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:22.358267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:22.358278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:22.359955 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360438 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:22.360442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:22.360443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:22.360553 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360855 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:22.360856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:22.360856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:22.361169 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:22.361238 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:22.361543 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:52:22.361690 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260619 02:52:22.361836 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.350791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:54013 (local address 127.26.98.2:35043)
0619 02:52:22.350993 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:22.351008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:22.351036 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:52:22.351761 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:22.351764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:22.351834 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:22.352008 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:22.352921 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.353868 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.354930 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.355341 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.355445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:22.357953 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:22.357960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:22.357961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:22.357975 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:22.359958 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360622 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:22.360625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:22.360625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:22.360718 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360905 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:22.360906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:22.360906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:22.361167 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:22.361238 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:22.361540 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:52:22.361692 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260619 02:52:22.361922 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.350591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:41135 (local address 127.26.98.1:52739)
0619 02:52:22.350726 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.350740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:22.350814 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:22.352095 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:22.352099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:22.352553 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:22.352796 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:22.352799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.353966 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.353967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:22.354811 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:22.354814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:22.355423 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:22.355424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:22.355537 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:22.356112 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:22.356122 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:22.358130 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:22.360063 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:22.360077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:22.360338 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:22.360663 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:22.360664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:22.360664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:22.360790 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:22.361248 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:22.361254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:22.361379 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 02:52:22.361474 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:52:22.361835 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":148,"thread_start_us":71,"threads_started":1}
W20260619 02:52:22.362967 9436 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
W20260619 02:52:22.363274 9433 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
I20260619 02:52:22.363329 9433 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d80def822f9d42a2ba2a81b5d0a1a33a; no voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6
I20260619 02:52:22.363449 9818 raft_consensus.cc:2726] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:22.387014 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.386789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:42579)
0619 02:52:22.386868 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.386948 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:52:22.417763 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.417486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:46839)
0619 02:52:22.417659 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.417706 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260619 02:52:22.569059 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.568678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:44965)
0619 02:52:22.568900 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.568990 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260619 02:52:23.387960 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.387638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:46747)
0619 02:52:23.387802 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.387887 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260619 02:52:23.418716 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.418420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:34331)
0619 02:52:23.418598 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.418655 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260619 02:52:23.570195 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.569852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:44003)
0619 02:52:23.570052 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.570110 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260619 02:52:24.273217 9828 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:24.273371 9828 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:24.273594 9828 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
W20260619 02:52:24.274091 9436 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
W20260619 02:52:24.274163 9433 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
I20260619 02:52:24.274266 9433 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d80def822f9d42a2ba2a81b5d0a1a33a; no voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6
I20260619 02:52:24.274389 9828 raft_consensus.cc:2726] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:24.389012 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.388698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:54529)
0619 02:52:24.388876 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.388928 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260619 02:52:24.419700 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.419407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:42859)
0619 02:52:24.419590 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.419640 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260619 02:52:24.571251 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.570962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:37117)
0619 02:52:24.571128 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.571181 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260619 02:52:25.390031 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:25.389676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:48661)
0619 02:52:25.389885 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:25.389964 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260619 02:52:25.420667 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:25.420328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:59745)
0619 02:52:25.420531 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:25.420600 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260619 02:52:25.572177 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:25.571862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:41831)
0619 02:52:25.572044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:25.572101 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260619 02:52:26.391292 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.391010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:56201)
0619 02:52:26.391166 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.391228 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260619 02:52:26.421732 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:42655)
0619 02:52:26.421621 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.421667 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260619 02:52:26.573222 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.572849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:54047)
0619 02:52:26.573089 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.573153 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260619 02:52:26.865823 9838 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:26.865955 9838 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:26.866329 9838 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
W20260619 02:52:26.866909 9433 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
W20260619 02:52:26.867013 9436 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
I20260619 02:52:26.867089 9436 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d80def822f9d42a2ba2a81b5d0a1a33a; no voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6
I20260619 02:52:26.867162 9838 raft_consensus.cc:2726] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:27.392253 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:27.391915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:33089)
0619 02:52:27.392111 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:27.392187 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260619 02:52:27.423000 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:27.422494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:39443)
0619 02:52:27.422788 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:27.422930 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260619 02:52:27.574275 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:27.573928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:50117)
0619 02:52:27.574153 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:27.574209 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260619 02:52:28.393256 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:28.392916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:41199)
0619 02:52:28.393118 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:28.393190 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260619 02:52:28.424052 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:28.423741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:39493)
0619 02:52:28.423934 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:28.423986 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260619 02:52:28.575338 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:28.575028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:60367)
0619 02:52:28.575211 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:28.575271 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260619 02:52:29.394251 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:29.393965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:34847)
0619 02:52:29.394133 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:29.394189 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260619 02:52:29.425028 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:29.424716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:57065)
0619 02:52:29.424909 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:29.424964 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260619 02:52:29.576292 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:29.575925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:51935)
0619 02:52:29.576147 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:29.576198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":119,"threads_started":1}
I20260619 02:52:29.921650 9848 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:29.921816 9848 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:29.922019 9848 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
W20260619 02:52:29.922619 9436 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
W20260619 02:52:29.922726 9433 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
I20260619 02:52:29.922777 9433 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d80def822f9d42a2ba2a81b5d0a1a33a; no voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6
I20260619 02:52:29.922853 9848 raft_consensus.cc:2726] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:30.395294 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:30.395007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:60129)
0619 02:52:30.395165 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:30.395229 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260619 02:52:30.426072 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:30.425746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:55273)
0619 02:52:30.425958 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:30.426008 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260619 02:52:30.577262 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:30.576927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:40795)
0619 02:52:30.577144 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:30.577199 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260619 02:52:31.396358 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:31.395991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:35171)
0619 02:52:31.396202 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:31.396285 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260619 02:52:31.427218 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:31.426757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:40269)
0619 02:52:31.427001 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:31.427071 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260619 02:52:31.578326 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:31.577995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:41659)
0619 02:52:31.578205 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:31.578260 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":121,"threads_started":1}
I20260619 02:52:32.397502 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:32.397149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:42319)
0619 02:52:32.397356 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:32.397425 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260619 02:52:32.428386 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:32.427994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:42921)
0619 02:52:32.428241 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:32.428312 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260619 02:52:32.579479 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:32.579135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:54679)
0619 02:52:32.579341 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:32.579406 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":130,"threads_started":1}
I20260619 02:52:33.398502 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:33.398143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:54919)
0619 02:52:33.398380 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:33.398438 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260619 02:52:33.429558 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:33.429223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:54767)
0619 02:52:33.429439 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:33.429491 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260619 02:52:33.580569 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:33.580281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:36405)
0619 02:52:33.580446 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:33.580504 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260619 02:52:33.669286 9861 raft_consensus.cc:482] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:33.669445 9861 raft_consensus.cc:504] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4749dc9ee18b4dafae4195f902057137" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 35043 } } peers { permanent_uuid: "5ad7dec64be141888303b1f697a54de6" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 41135 } } peers { permanent_uuid: "d80def822f9d42a2ba2a81b5d0a1a33a" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 42079 } }
I20260619 02:52:33.669757 9861 leader_election.cc:288] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043), 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135)
W20260619 02:52:33.670308 9436 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4749dc9ee18b4dafae4195f902057137 (127.26.98.2:35043): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
W20260619 02:52:33.670403 9433 leader_election.cc:341] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad7dec64be141888303b1f697a54de6 (127.26.98.3:41135): Not found: Tablet not found: 99d652c42297464c96e49af262801da5
I20260619 02:52:33.670462 9433 leader_election.cc:302] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d80def822f9d42a2ba2a81b5d0a1a33a; no voters: 4749dc9ee18b4dafae4195f902057137, 5ad7dec64be141888303b1f697a54de6
I20260619 02:52:33.670547 9861 raft_consensus.cc:2726] T 99d652c42297464c96e49af262801da5 P d80def822f9d42a2ba2a81b5d0a1a33a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:34.399601 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:34.399265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:42701)
0619 02:52:34.399447 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:34.399536 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260619 02:52:34.430928 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:34.430416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:50907)
0619 02:52:34.430714 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:34.430776 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":189,"threads_started":1}
I20260619 02:52:34.581753 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:34.581370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:40895)
0619 02:52:34.581570 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:34.581685 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":158,"threads_started":1}
I20260619 02:52:35.400647 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:35.400380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:39925)
0619 02:52:35.400539 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:35.400583 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260619 02:52:35.431993 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:35.431588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:58789)
0619 02:52:35.431844 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:35.431921 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260619 02:52:35.582856 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:35.582515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:56843)
0619 02:52:35.582725 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:35.582788 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260619 02:52:36.401835 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:36.401463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.1:38733)
0619 02:52:36.401690 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:36.401759 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260619 02:52:36.433025 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:36.432711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.2:40733)
0619 02:52:36.432905 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:36.432963 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
I20260619 02:52:36.584131 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:36.583713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:35015 (local address 127.26.98.3:40217)
0619 02:52:36.583986 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:36.584064 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:35015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260619 02:52:36.586393 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9388
I20260619 02:52:36.593266 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9551
I20260619 02:52:36.598631 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9685
2026-06-19T02:52:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18822 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:36 dist-test-slave-0f9d krb5kdc[9873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:36 dist-test-slave-0f9d krb5kdc[9873](info): set up 2 sockets
Jun 19 02:52:36 dist-test-slave-0f9d krb5kdc[9873](info): commencing operation
krb5kdc: starting...
W20260619 02:52:37.607805 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:52:37 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:52:37.746129 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:37.846478 9886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:37.847035 9886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:37.847080 9886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:37.848865 9886 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:52:37.848955 9886 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:52:37.848969 9886 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:52:37.848994 9886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:37.849010 9886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:37.849025 9886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:37.849037 9886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:37.849049 9886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:37.852064 9886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:37.852393 9886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:37.852682 9886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:37.859280 9894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:37.859256 9892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:37.859305 9886 server_base.cc:1034] running on GCE node
W20260619 02:52:37.859256 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:37.859891 9886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:37.860374 9886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:37.861589 9886 hybrid_clock.cc:648] HybridClock initialized: now 1781837557861571 us; error 52 us; skew 500 ppm
Jun 19 02:52:37 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837557, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:37.865725 9886 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:37.866618 9886 webserver.cc:460] Webserver started at http://127.26.98.62:35839/ using document root <none> and password file <none>
I20260619 02:52:37.867000 9886 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:37.867201 9886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:37.867408 9886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:37.869123 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "338a0248b1c846449ab56cdd9d855328"
format_stamp: "Formatted at 2026-06-19 02:52:37 on dist-test-slave-0f9d"
server_key: "32b75c931f0201a22f1f0d737cb145a7"
server_key_iv: "29b7764cbff554e790eb28b3f314d60a"
server_key_version: "encryptionkey@0"
I20260619 02:52:37.869645 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "338a0248b1c846449ab56cdd9d855328"
format_stamp: "Formatted at 2026-06-19 02:52:37 on dist-test-slave-0f9d"
server_key: "32b75c931f0201a22f1f0d737cb145a7"
server_key_iv: "29b7764cbff554e790eb28b3f314d60a"
server_key_version: "encryptionkey@0"
I20260619 02:52:37.871505 9886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:52:37.872728 9901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:37.873138 9886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:37.873267 9886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "338a0248b1c846449ab56cdd9d855328"
format_stamp: "Formatted at 2026-06-19 02:52:37 on dist-test-slave-0f9d"
server_key: "32b75c931f0201a22f1f0d737cb145a7"
server_key_iv: "29b7764cbff554e790eb28b3f314d60a"
server_key_version: "encryptionkey@0"
I20260619 02:52:37.873370 9886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:37.887734 9886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:37.890976 9886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:37.891166 9886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:37.896554 9886 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45487
I20260619 02:52:37.896857 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45487 every 8 connection(s)
I20260619 02:52:37.897058 9886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:37.898156 9954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:37.901391 9954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap starting.
I20260619 02:52:37.902225 9954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:37.902573 9954 log.cc:824] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:37.903647 9954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: No bootstrap required, opened a new log
I20260619 02:52:37.903661 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9886
I20260619 02:52:37.903784 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:52:37.904047 27016 external_mini_cluster.cc:1428] Setting key 189d76b935282b8805352759569b6f8d
I20260619 02:52:37.906100 9954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:37.906383 9954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:37.906425 9954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Initialized, Role: FOLLOWER
I20260619 02:52:37.906587 9954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:37.906677 9954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:37.906716 9954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:37.906778 9954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:37.907693 9954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:37.907852 9954 leader_election.cc:302] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338a0248b1c846449ab56cdd9d855328; no voters:
I20260619 02:52:37.908102 9954 leader_election.cc:288] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:37.908178 9959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:37.908344 9959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 LEADER]: Becoming Leader. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Running, Role: LEADER
I20260619 02:52:37.908501 9954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:37.908547 9959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:37.909097 9960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:52:37.909204 9960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:37.909502 9961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338a0248b1c846449ab56cdd9d855328. Latest consensus state: current_term: 1 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:52:37.909621 9961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:37.910019 9967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:37.910861 9967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:37.914187 9967 catalog_manager.cc:1347] Generated new cluster ID: 2db7a836b75e4d359411617945888f74
I20260619 02:52:37.914304 9967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:52:37 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:37.921305 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:37.904671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.26.98.62:45487)
0619 02:52:37.904977 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:37.904989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:37.905027 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:52:37.906109 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:37.906128 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:37.906229 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:37.906407 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:37.907330 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:37.909355 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:37.910834 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:37.911563 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:37.911716 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:37.915553 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:37.915560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:37.915560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:37.915579 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:37.918720 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:37.919450 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:37.919455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:37.919455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:37.919727 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:37.920001 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:37.920007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:37.920007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:37.920446 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:37.920636 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:37.920766 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:52:37.920966 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":155,"threads_started":1}
I20260619 02:52:37.928300 9967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:37.929303 9967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:37.935801 9967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Generated new TSK 0
I20260619 02:52:37.936134 9967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:52:37.980254 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:38.084017 9980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:38.084314 9980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:38.084347 9980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:38.086374 9980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:38.086459 9980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:38.086498 9980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:38.089321 9980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:38.089860 9980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:38.090222 9980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:38.096804 9985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:38.096841 9986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:38.096905 9980 server_base.cc:1034] running on GCE node
W20260619 02:52:38.096804 9988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:38.097272 9980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:38.097620 9980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:38.098835 9980 hybrid_clock.cc:648] HybridClock initialized: now 1781837558098809 us; error 38 us; skew 500 ppm
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:38.103096 9980 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:38.103753 9980 webserver.cc:460] Webserver started at http://127.26.98.1:46745/ using document root <none> and password file <none>
I20260619 02:52:38.104000 9980 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:38.104051 9980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:38.104172 9980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:38.105543 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "32eded6e0647efa8ee412e9887289b6f"
server_key_iv: "a91e5f4e06b5234ee4546c3673cf4ee6"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.105964 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "32eded6e0647efa8ee412e9887289b6f"
server_key_iv: "a91e5f4e06b5234ee4546c3673cf4ee6"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.107905 9980 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:52:38.109054 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.109277 9980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:38.109385 9980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "32eded6e0647efa8ee412e9887289b6f"
server_key_iv: "a91e5f4e06b5234ee4546c3673cf4ee6"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.109650 9980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:38.135628 9980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:38.138962 9980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:38.139150 9980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:38.139483 9980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:38.139879 9980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:38.139928 9980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.139958 9980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:38.139969 9980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.149662 9980 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36235
I20260619 02:52:38.149782 10108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36235 every 8 connection(s)
I20260619 02:52:38.150403 9980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:38.157966 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 9980
I20260619 02:52:38.158106 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:52:38.158325 27016 external_mini_cluster.cc:1428] Setting key 18c7c7442c6dc582c46b04b2ad02b145
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:38.163734 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.151545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:36149 (local address 127.26.98.62:45487)
0619 02:52:38.151640 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.151642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.151925 (+ 283us) server_negotiation.cc:407] Connection header received
0619 02:52:38.152612 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.152614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.152629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.152691 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:38.153968 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.154546 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.155814 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.156061 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.156145 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.159201 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:38.159203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:38.159203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:38.159212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:38.161407 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.162084 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.162086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.162087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.162183 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.162524 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.162525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.162526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.162702 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:38.162775 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.163448 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.163601 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:52:38.164019 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.151534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:36149)
0619 02:52:38.151758 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.151806 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.152484 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.152790 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.152795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.153544 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:38.153831 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.153838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.154657 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.154658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.155697 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.155700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.156163 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.156164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.156349 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.157227 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:38.157238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:38.159084 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:38.161546 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.161561 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.161566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.161987 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.162270 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.162271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.162272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.162455 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.162852 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:38.162870 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:38.163158 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.163316 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.163715 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":124,"threads_started":1}
I20260619 02:52:38.164984 10109 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:38.165148 10109 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:38.165432 10109 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:52:38.170655 9918 ts_manager.cc:194] Registered new tserver with Master: 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
I20260619 02:52:38.172398 9918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:36149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:52:38.204645 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:38.308126 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:38.308488 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:38.308526 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:38.310302 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:38.310397 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:38.310437 10114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:38.312759 10114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:38.313117 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:38.313400 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:38.320000 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:38.320005 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:38.320005 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:38.320523 10114 server_base.cc:1034] running on GCE node
I20260619 02:52:38.320853 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:38.321171 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:38.322355 10114 hybrid_clock.cc:648] HybridClock initialized: now 1781837558322317 us; error 58 us; skew 500 ppm
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:38.325821 10114 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:38.326479 10114 webserver.cc:460] Webserver started at http://127.26.98.2:46505/ using document root <none> and password file <none>
I20260619 02:52:38.326809 10114 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:38.326916 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:38.327095 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:38.328444 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "d37405c1845dc6abec1efccd98288901"
server_key_iv: "54ea529dbc88322afa1c4ca82ae4b375"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.328851 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "d37405c1845dc6abec1efccd98288901"
server_key_iv: "54ea529dbc88322afa1c4ca82ae4b375"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.330482 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:52:38.331475 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.331655 10114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:38.331799 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "d37405c1845dc6abec1efccd98288901"
server_key_iv: "54ea529dbc88322afa1c4ca82ae4b375"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.331897 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:38.344399 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:38.347608 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:38.347795 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:38.348119 10114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:38.348508 10114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:38.348547 10114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.348572 10114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:38.348584 10114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.356462 10114 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:46507
I20260619 02:52:38.356513 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:46507 every 8 connection(s)
I20260619 02:52:38.357074 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:38.363415 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10114
I20260619 02:52:38.363549 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:52:38.363811 27016 external_mini_cluster.cc:1428] Setting key f95e2febae77ec81c634d6e7b202a32b
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:38.371876 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.358199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:50743 (local address 127.26.98.62:45487)
0619 02:52:38.358270 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.358272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.358466 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:52:38.359125 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.359127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.359140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.359208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:38.360441 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.361082 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.362633 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.362991 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.363064 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.366386 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:38.366389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:38.366389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:38.366405 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:38.368991 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.370088 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.370090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.370091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.370205 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.370790 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.370791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.370792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.371030 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:38.371111 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.371628 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.371753 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:52:38.372166 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.358070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:50743)
0619 02:52:38.358274 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.358328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.358990 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.359289 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.359294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.360006 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:38.360291 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.360298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.361297 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.361299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.362492 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.362498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.363098 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.363099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.363285 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.364035 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:38.364047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:38.366232 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:38.369158 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.369176 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.369180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.369838 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.370407 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.370409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.370410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.370649 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.371122 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:38.371129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:38.371367 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.371598 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.371865 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260619 02:52:38.372998 10243 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:38.373157 10243 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:38.373423 10243 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
I20260619 02:52:38.374073 9918 ts_manager.cc:194] Registered new tserver with Master: 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507)
I20260619 02:52:38.374758 9918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:50743
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:52:38.400002 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:38.502954 10248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:38.503268 10248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:38.503314 10248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:38.504902 10248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:38.504972 10248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:38.505019 10248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:38.506964 10248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:38.507306 10248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:38.507571 10248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:38.513660 10253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:38.513633 10254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:38.513633 10256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:38.513876 10248 server_base.cc:1034] running on GCE node
I20260619 02:52:38.514142 10248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:38.514506 10248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:38.515678 10248 hybrid_clock.cc:648] HybridClock initialized: now 1781837558515663 us; error 45 us; skew 500 ppm
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:38.519405 10248 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:38.520058 10248 webserver.cc:460] Webserver started at http://127.26.98.3:35503/ using document root <none> and password file <none>
I20260619 02:52:38.520318 10248 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:38.520376 10248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:38.520525 10248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:38.521970 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "0cb80e71069f4916a05359a44773ceef"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "89af67c55ab035ac73ac5b4353db8a16"
server_key_iv: "423ee106ae89ff4cea0f342c0e251cfe"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.522403 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "0cb80e71069f4916a05359a44773ceef"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "89af67c55ab035ac73ac5b4353db8a16"
server_key_iv: "423ee106ae89ff4cea0f342c0e251cfe"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.524569 10248 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:52:38.525828 10263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.526274 10248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:38.526405 10248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "0cb80e71069f4916a05359a44773ceef"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "89af67c55ab035ac73ac5b4353db8a16"
server_key_iv: "423ee106ae89ff4cea0f342c0e251cfe"
server_key_version: "encryptionkey@0"
I20260619 02:52:38.526513 10248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:38.558069 10248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:38.560832 10248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:38.561021 10248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:38.561324 10248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:38.561707 10248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:38.561744 10248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.561782 10248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:38.561800 10248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:38.570456 10248 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:36497
I20260619 02:52:38.570506 10376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:36497 every 8 connection(s)
I20260619 02:52:38.571007 10248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:38.578387 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10248
I20260619 02:52:38.578517 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:52:38.578750 27016 external_mini_cluster.cc:1428] Setting key a3854def709a1f865986716979f1a03c
Jun 19 02:52:38 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837558, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:38.585050 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.572001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:60517 (local address 127.26.98.62:45487)
0619 02:52:38.572109 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.572112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.572307 (+ 195us) server_negotiation.cc:407] Connection header received
0619 02:52:38.573321 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.573323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.573340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.573516 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:38.575074 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.575660 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.576996 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.577248 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.577315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.580856 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:38.580858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:38.580859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:38.580870 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:38.582605 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.583301 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.583302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.583303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.583442 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.584008 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.584010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.584010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.584205 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:38.584369 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.584788 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.584965 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:52:38.585188 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.571909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:60517)
0619 02:52:38.572156 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.572208 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.573087 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.573645 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.573651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.574569 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:38.574873 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.574909 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.575792 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.575794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.576876 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.576882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.577344 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.577346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.577526 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.578273 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:38.578288 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:38.580700 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:38.582752 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.582772 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.582775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.583212 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.583549 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:38.583550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:38.583550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:38.583710 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:38.584356 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:38.584366 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:38.584572 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.584685 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.584908 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260619 02:52:38.585801 10377 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:38.585932 10377 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:38.586143 10377 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
I20260619 02:52:38.586637 9918 ts_manager.cc:194] Registered new tserver with Master: 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497)
I20260619 02:52:38.587118 9918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:60517
I20260619 02:52:38.591936 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:38.601841 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.593846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.26.98.62:45487)
0619 02:52:38.593922 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.593924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.594001 (+ 77us) server_negotiation.cc:407] Connection header received
0619 02:52:38.594089 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.594090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.594103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.594168 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:38.594978 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.595564 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.596413 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.596622 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.596687 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.598395 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:38.598397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:38.598397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:38.598408 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:38.600326 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.600850 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.600851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.600852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.600917 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:38.601110 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:38.601111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:38.601111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:38.601316 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:38.601435 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.601563 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.601682 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:52:38.604296 9918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:38.606562 9918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:38.615417 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.26.98.1:36235)
0619 02:52:38.610941 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.610946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.610957 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:38.611007 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.611010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.611027 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.611401 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:38.611537 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.612640 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.614287 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.614944 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.615016 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.615089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.615266 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.615310 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":59,"threads_started":1}
I20260619 02:52:38.615406 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.0.0.1:55752)
0619 02:52:38.610458 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.610506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.610576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.611241 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.611242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.611244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:38.611285 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.611287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.612753 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.612754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.614134 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.614140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.615043 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.615045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.615163 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.615176 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.615241 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.615317 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":151,"threads_started":1}
I20260619 02:52:38.615942 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36497 (local address 127.0.0.1:49790)
0619 02:52:38.610695 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.610707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.610774 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.611115 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.611119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.611122 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:38.611233 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.611236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.612626 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.612629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.614175 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.614181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.615603 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.615604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.615674 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.615677 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.615819 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.615864 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":219,"thread_start_us":109,"threads_started":1}
I20260619 02:52:38.616379 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.26.98.3:36497)
0619 02:52:38.610905 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.610909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.610921 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:38.610973 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.610975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.610995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.611314 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:38.611400 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.612390 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.614309 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.615415 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.615508 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.615567 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.616226 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.616286 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260619 02:52:38.616753 10043 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 (DEFAULT_TABLE table=test-table [id=2babdeecffa94d8f9e39b858e89af6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:38.617012 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.0.0.1:41444)
0619 02:52:38.610853 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.610862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.610937 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.611217 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.611217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.611220 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:38.611284 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.611287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.614278 (+ 2991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.614279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.615515 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.615519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.616786 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.616788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.616854 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.616858 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.616907 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.616952 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":26,"thread_start_us":133,"threads_started":1}
I20260619 02:52:38.617331 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed5ea7fbf8455194623272e1cbea79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:38.617573 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.610482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.26.98.2:46507)
0619 02:52:38.610691 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.610697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.610993 (+ 296us) server_negotiation.cc:407] Connection header received
0619 02:52:38.611039 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.611041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.611061 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.611132 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:38.611361 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.612603 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.615941 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.616696 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.616769 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.616824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.617371 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.617467 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":100,"threads_started":1}
I20260619 02:52:38.618340 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 (DEFAULT_TABLE table=test-table [id=2babdeecffa94d8f9e39b858e89af6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:38.618706 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed5ea7fbf8455194623272e1cbea79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:38.619756 10311 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 (DEFAULT_TABLE table=test-table [id=2babdeecffa94d8f9e39b858e89af6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:38.620203 10311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed5ea7fbf8455194623272e1cbea79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:38.621050 10394 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap starting.
I20260619 02:52:38.621919 10394 tablet_bootstrap.cc:654] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:38.622483 10394 log.cc:824] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:38.622933 10396 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Bootstrap starting.
I20260619 02:52:38.623401 10394 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: No bootstrap required, opened a new log
I20260619 02:52:38.623472 10394 ts_tablet_manager.cc:1397] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:38.624915 10397 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Bootstrap starting.
I20260619 02:52:38.625288 10394 raft_consensus.cc:348] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.625512 10394 raft_consensus.cc:374] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:38.625563 10394 raft_consensus.cc:729] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3019050d32a14bfb83d03c5ee4ff6aab, State: Initialized, Role: FOLLOWER
I20260619 02:52:38.625713 10394 consensus_queue.cc:260] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.625933 10397 tablet_bootstrap.cc:654] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:38.626050 10394 ts_tablet_manager.cc:1428] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:38.626075 10109 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
I20260619 02:52:38.627022 10396 tablet_bootstrap.cc:654] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:38.627568 10397 log.cc:824] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:38.627769 10396 log.cc:824] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:38.628516 10396 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: No bootstrap required, opened a new log
I20260619 02:52:38.628520 10397 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: No bootstrap required, opened a new log
I20260619 02:52:38.628588 10396 ts_tablet_manager.cc:1397] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 02:52:38.628592 10397 ts_tablet_manager.cc:1397] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:52:38.630514 10396 raft_consensus.cc:348] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.630738 10396 raft_consensus.cc:374] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:38.630772 10396 raft_consensus.cc:729] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885bbfb8247d4d04b7c0a07885a7a3f6, State: Initialized, Role: FOLLOWER
I20260619 02:52:38.630950 10396 consensus_queue.cc:260] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.631439 10396 ts_tablet_manager.cc:1428] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 02:52:38.631511 10243 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
I20260619 02:52:38.631704 10397 raft_consensus.cc:348] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.631963 10397 raft_consensus.cc:374] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:38.632002 10397 raft_consensus.cc:729] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb80e71069f4916a05359a44773ceef, State: Initialized, Role: FOLLOWER
I20260619 02:52:38.632172 10397 consensus_queue.cc:260] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.632606 10397 ts_tablet_manager.cc:1428] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 02:52:38.632735 10377 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
W20260619 02:52:38.651594 10110 tablet.cc:2369] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:38.821702 10378 tablet.cc:2369] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:38.858060 10244 tablet.cc:2369] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:38.872920 10398 raft_consensus.cc:482] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:38.873111 10398 raft_consensus.cc:504] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.873517 10398 leader_election.cc:288] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507), 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497)
I20260619 02:52:38.877838 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.873849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:41795 (local address 127.26.98.2:46507)
0619 02:52:38.873902 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.873904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.874065 (+ 161us) server_negotiation.cc:407] Connection header received
0619 02:52:38.874122 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.874123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.874138 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.874192 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:38.874362 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.874950 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.876612 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.877283 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.877377 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.877394 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.877672 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.877769 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:52:38.877835 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.873922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:43443 (local address 127.26.98.3:36497)
0619 02:52:38.873979 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.873981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.873991 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:38.874048 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.874049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.874063 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.874121 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:38.874285 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.874799 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.876574 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.877303 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.877377 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.877394 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.877672 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.877748 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:52:38.877852 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.873745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.26.98.1:41795)
0619 02:52:38.873981 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.873991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.874062 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.874202 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.874203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.874205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:38.874301 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.874303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.875035 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.875037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.876367 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.876372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.877423 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.877424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.877550 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.877559 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.877657 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.877701 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":6,"thread_start_us":146,"threads_started":1}
I20260619 02:52:38.877854 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.873735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36497 (local address 127.26.98.1:43443)
0619 02:52:38.873888 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.873902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:38.873977 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:38.874129 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:38.874130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:38.874133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:38.874228 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.874230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.874922 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.874923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.876366 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:38.876372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.877424 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.877425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:38.877550 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.877558 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:52:38.877628 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:52:38.877701 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":117,"thread_start_us":74,"threads_started":1}
I20260619 02:52:38.878409 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed5ea7fbf8455194623272e1cbea79" candidate_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" is_pre_election: true
I20260619 02:52:38.878409 10331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed5ea7fbf8455194623272e1cbea79" candidate_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb80e71069f4916a05359a44773ceef" is_pre_election: true
I20260619 02:52:38.878638 10197 raft_consensus.cc:2454] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3019050d32a14bfb83d03c5ee4ff6aab in term 0.
I20260619 02:52:38.878638 10331 raft_consensus.cc:2454] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3019050d32a14bfb83d03c5ee4ff6aab in term 0.
I20260619 02:52:38.879006 9998 leader_election.cc:302] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3019050d32a14bfb83d03c5ee4ff6aab, 885bbfb8247d4d04b7c0a07885a7a3f6; no voters:
I20260619 02:52:38.879276 10398 raft_consensus.cc:2781] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:38.879364 10398 raft_consensus.cc:482] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:38.879382 10398 raft_consensus.cc:3037] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:38.880338 10398 raft_consensus.cc:504] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.880525 10398 leader_election.cc:288] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [CANDIDATE]: Term 1 election: Requested vote from peers 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507), 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497)
I20260619 02:52:38.880818 10331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed5ea7fbf8455194623272e1cbea79" candidate_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb80e71069f4916a05359a44773ceef"
I20260619 02:52:38.880894 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed5ea7fbf8455194623272e1cbea79" candidate_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
I20260619 02:52:38.880932 10331 raft_consensus.cc:3037] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:38.881003 10197 raft_consensus.cc:3037] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:38.881892 10197 raft_consensus.cc:2454] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3019050d32a14bfb83d03c5ee4ff6aab in term 1.
I20260619 02:52:38.881910 10331 raft_consensus.cc:2454] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3019050d32a14bfb83d03c5ee4ff6aab in term 1.
I20260619 02:52:38.882166 9999 leader_election.cc:302] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb80e71069f4916a05359a44773ceef, 3019050d32a14bfb83d03c5ee4ff6aab; no voters:
I20260619 02:52:38.882513 10398 raft_consensus.cc:2781] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:38.882762 10398 raft_consensus.cc:686] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 LEADER]: Becoming Leader. State: Replica: 3019050d32a14bfb83d03c5ee4ff6aab, State: Running, Role: LEADER
I20260619 02:52:38.882920 10398 consensus_queue.cc:237] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:38.884027 9917 catalog_manager.cc:5495] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1). New cstate: current_term: 1 leader_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } health_report { overall_health: HEALTHY } } }
I20260619 02:52:38.910239 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.906859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.26.98.1:36235)
0619 02:52:38.906979 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:38.906981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:38.907014 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:52:38.907062 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:38.907063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:38.907089 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:38.907168 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:38.907456 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.908011 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.909059 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:38.909332 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:38.909424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:38.909752 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:38.909950 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:38.910045 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:38.910091 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 02:52:38.910122 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:52:38.912734 10197 raft_consensus.cc:1261] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Refusing update from remote peer 3019050d32a14bfb83d03c5ee4ff6aab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:38.912773 10331 raft_consensus.cc:1261] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Refusing update from remote peer 3019050d32a14bfb83d03c5ee4ff6aab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:38.913234 10405 consensus_queue.cc:1035] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:38.913451 10398 consensus_queue.cc:1035] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [LEADER]: Connected to new peer: Peer: permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:38.916909 10408 mvcc.cc:205] Tried to move back new op lower bound from 7298406641303175168 to 7298406641185468416. Current Snapshot: MvccSnapshot[applied={T|T < 7298406641303175168}]
I20260619 02:52:38.918169 10410 mvcc.cc:205] Tried to move back new op lower bound from 7298406641303175168 to 7298406641185468416. Current Snapshot: MvccSnapshot[applied={T|T < 7298406641303175168 or (T in {7298406641303175168})}]
I20260619 02:52:38.920403 9916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:38.920533 9916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:38.921785 9916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Sending DeleteTablet for 3 replicas of tablet 7fed5ea7fbf8455194623272e1cbea79
I20260619 02:52:38.922224 10043 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:38 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:55752
I20260619 02:52:38.922250 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:38 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:41444
I20260619 02:52:38.922430 10311 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:38 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:49790
I20260619 02:52:38.922626 10418 tablet_replica.cc:330] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: stopping tablet replica
I20260619 02:52:38.922777 10418 raft_consensus.cc:2229] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:38.922806 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9980
I20260619 02:52:38.924716 10418 raft_consensus.cc:2258] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:38.925061 10419 tablet_replica.cc:330] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: stopping tablet replica
I20260619 02:52:38.925230 10419 raft_consensus.cc:2229] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:38.925369 10419 raft_consensus.cc:2258] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:38.925734 10418 ts_tablet_manager.cc:1905] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:38.925916 10419 ts_tablet_manager.cc:1905] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:38.928647 10419 ts_tablet_manager.cc:1918] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:38.928647 10418 ts_tablet_manager.cc:1918] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:38.928755 10419 log.cc:1196] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/7fed5ea7fbf8455194623272e1cbea79
I20260619 02:52:38.928825 10418 log.cc:1196] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/7fed5ea7fbf8455194623272e1cbea79
W20260619 02:52:38.928918 9902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:36235 (error 108)
I20260619 02:52:38.929335 10419 ts_tablet_manager.cc:1939] T 7fed5ea7fbf8455194623272e1cbea79 P 0cb80e71069f4916a05359a44773ceef: Deleting consensus metadata
I20260619 02:52:38.929656 10418 ts_tablet_manager.cc:1939] T 7fed5ea7fbf8455194623272e1cbea79 P 885bbfb8247d4d04b7c0a07885a7a3f6: Deleting consensus metadata
I20260619 02:52:38.929677 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.929416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.0.0.1:55770)
0619 02:52:38.929476 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.929588 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 02:52:38.929874 9902 catalog_manager.cc:4584] TS 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fed5ea7fbf8455194623272e1cbea79: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36235: connect: Connection refused (error 111)
I20260619 02:52:38.930092 9905 catalog_manager.cc:4841] TS 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497): tablet 7fed5ea7fbf8455194623272e1cbea79 (table test-table [id=2babdeecffa94d8f9e39b858e89af6e2]) successfully deleted
I20260619 02:52:38.930255 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10114
I20260619 02:52:38.935762 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10248
I20260619 02:52:38.935969 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:38.935816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.0.0.1:41456)
0619 02:52:38.935875 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:38.935922 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 02:52:38.940783 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 9886
I20260619 02:52:58.947338 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--webserver_interface=127.26.98.62
--webserver_port=35839
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:59.048300 10421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:59.048607 10421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:59.048774 10421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:59.051018 10421 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:52:59.051095 10421 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:52:59.051110 10421 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:52:59.051122 10421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:59.051137 10421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:59.051151 10421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:59.051163 10421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:59.051177 10421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:59.053157 10421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:59.053437 10421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:59.053659 10421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:59.065069 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.065160 10421 server_base.cc:1034] running on GCE node
W20260619 02:52:59.065178 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.065393 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.065758 10421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:59.066164 10421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:59.067360 10421 hybrid_clock.cc:648] HybridClock initialized: now 1781837579067310 us; error 64 us; skew 500 ppm
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:59.070859 10421 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:52:59.071496 10421 webserver.cc:460] Webserver started at http://127.26.98.62:35839/ using document root <none> and password file <none>
I20260619 02:52:59.071790 10421 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:59.071866 10421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:59.073606 10421 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:59.074857 10437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.075102 10421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:59.075213 10421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "338a0248b1c846449ab56cdd9d855328"
format_stamp: "Formatted at 2026-06-19 02:52:37 on dist-test-slave-0f9d"
server_key: "32b75c931f0201a22f1f0d737cb145a7"
server_key_iv: "29b7764cbff554e790eb28b3f314d60a"
server_key_version: "encryptionkey@0"
I20260619 02:52:59.075806 10421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:59.100365 10421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:59.103838 10421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:59.104049 10421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:59.115000 10421 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45487
I20260619 02:52:59.115487 10489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45487 every 8 connection(s)
I20260619 02:52:59.115662 10421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:59.115967 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10421
I20260619 02:52:59.116328 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:36235
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=46745
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:59.117571 10490 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:59.118732 10490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap starting.
I20260619 02:52:59.140102 10490 log.cc:824] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:59.143586 10490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:59.144109 10490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap complete.
I20260619 02:52:59.148721 10490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:59.149478 10490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Initialized, Role: FOLLOWER
I20260619 02:52:59.149827 10490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:59.149959 10490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:59.150020 10490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:59.150069 10490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:52:59.152696 10490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:59.152913 10490 leader_election.cc:302] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338a0248b1c846449ab56cdd9d855328; no voters:
I20260619 02:52:59.153156 10490 leader_election.cc:288] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:52:59.153326 10494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:52:59.153494 10490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:59.153718 10494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 LEADER]: Becoming Leader. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Running, Role: LEADER
I20260619 02:52:59.153860 10494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:52:59.154397 10495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:52:59.154486 10496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338a0248b1c846449ab56cdd9d855328. Latest consensus state: current_term: 2 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:52:59.154506 10495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:59.154596 10496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:59.154798 10503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:59.155887 10503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:59.156044 10503 catalog_manager.cc:1259] Loaded cluster ID: 2db7a836b75e4d359411617945888f74
I20260619 02:52:59.156095 10503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:59.157580 10503 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:59.157866 10503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Loaded TSK: 0
I20260619 02:52:59.172247 10503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Generated new TSK 1
I20260619 02:52:59.172453 10503 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:59.248402 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:59.248978 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:59.249030 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:59.251534 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:59.251645 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:59.251703 10492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:52:59.254222 10492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:36235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=46745
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:59.254657 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:59.255033 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:59.262135 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.262137 10518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.262137 10520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.263758 10492 server_base.cc:1034] running on GCE node
I20260619 02:52:59.264113 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:59.265439 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:59.267398 10492 hybrid_clock.cc:648] HybridClock initialized: now 1781837579267496 us; error 188 us; skew 500 ppm
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:59.272579 10492 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:52:59.273520 10492 webserver.cc:460] Webserver started at http://127.26.98.1:46745/ using document root <none> and password file <none>
I20260619 02:52:59.273940 10492 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:59.274029 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:59.275897 10492 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:59.276728 10527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.276962 10492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:59.277055 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "32eded6e0647efa8ee412e9887289b6f"
server_key_iv: "a91e5f4e06b5234ee4546c3673cf4ee6"
server_key_version: "encryptionkey@0"
I20260619 02:52:59.277544 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:59.289950 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:59.293915 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:59.294106 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:59.294519 10492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:59.295687 10534 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:59.298502 10492 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:59.298612 10492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.001s
I20260619 02:52:59.298662 10492 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:59.299633 10492 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:59.299679 10492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:59.299808 10534 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap starting.
I20260619 02:52:59.310130 10492 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36235
I20260619 02:52:59.310684 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:59.311437 10641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36235 every 8 connection(s)
I20260619 02:52:59.315632 10534 log.cc:824] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:59.316675 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10492
I20260619 02:52:59.317103 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:46507
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=46505
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:59.319715 10534 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:59.320206 10534 tablet_bootstrap.cc:492] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap complete.
I20260619 02:52:59.321642 10534 ts_tablet_manager.cc:1397] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:59.323565 10534 raft_consensus.cc:348] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.324054 10534 raft_consensus.cc:729] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3019050d32a14bfb83d03c5ee4ff6aab, State: Initialized, Role: FOLLOWER
I20260619 02:52:59.324339 10534 consensus_queue.cc:260] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.324841 10534 ts_tablet_manager.cc:1428] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:52:59.329440 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.311802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:48341 (local address 127.26.98.62:45487)
0619 02:52:59.313005 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.313016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.313051 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:52:59.313200 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.313205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.313255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.313405 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:59.314344 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.315902 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.318509 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.319751 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.320791 (+ 1040us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.324656 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:59.324665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:59.324665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:59.324690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:59.327097 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.327740 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.327743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.327744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.327877 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.328215 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.328216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.328216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.328535 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:59.328622 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.328991 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.329168 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":80,"threads_started":1}
I20260619 02:52:59.329428 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.311703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:48341)
0619 02:52:59.311983 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.312047 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.312740 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.313507 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.313512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.313888 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:59.314150 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.314158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.316346 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.316349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.318238 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.318246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.320023 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.320026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.320373 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.321377 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:59.321392 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:59.324410 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:59.327281 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.327297 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.327300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.327633 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.327985 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.327987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.327987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.328130 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.328647 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:59.328656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:59.328807 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.328910 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.329168 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":136,"threads_started":1}
I20260619 02:52:59.330412 10642 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:59.330539 10642 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:59.330776 10642 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
I20260619 02:52:59.331739 10454 ts_manager.cc:194] Registered new tserver with Master: 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
I20260619 02:52:59.333046 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:48341
I20260619 02:52:59.334307 10642 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
I20260619 02:52:59.340281 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.332560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.0.0.1:45932)
0619 02:52:59.335105 (+ 2545us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.335167 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.335261 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.335464 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.335468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.335471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:59.335595 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.335597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.336532 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.336534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.338305 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.338316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.339659 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.339661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.339885 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.339917 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.340038 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.340142 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2496,"thread_start_us":151,"threads_started":1}
I20260619 02:52:59.340281 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.333434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.26.98.1:36235)
0619 02:52:59.333653 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.333660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.335253 (+ 1593us) server_negotiation.cc:407] Connection header received
0619 02:52:59.335354 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.335357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.335376 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.335460 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:59.335666 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.336417 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.338499 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.339444 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.339634 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.339747 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.340095 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.340168 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260619 02:52:59.341279 10576 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed5ea7fbf8455194623272e1cbea79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:38 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45932
I20260619 02:52:59.343104 10651 tablet_replica.cc:330] stopping tablet replica
I20260619 02:52:59.343318 10651 raft_consensus.cc:2229] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:59.343415 10651 raft_consensus.cc:2258] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:59.343855 10651 ts_tablet_manager.cc:1905] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:59.346844 10651 ts_tablet_manager.cc:1918] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:59.347024 10651 log.cc:1196] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal/wals/7fed5ea7fbf8455194623272e1cbea79
I20260619 02:52:59.347297 10651 ts_tablet_manager.cc:1939] T 7fed5ea7fbf8455194623272e1cbea79 P 3019050d32a14bfb83d03c5ee4ff6aab: Deleting consensus metadata
I20260619 02:52:59.348158 10438 catalog_manager.cc:4841] TS 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235): tablet 7fed5ea7fbf8455194623272e1cbea79 (table test-table [id=2babdeecffa94d8f9e39b858e89af6e2]) successfully deleted
W20260619 02:52:59.440827 10647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:59.441118 10647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:59.441147 10647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:59.442561 10647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:59.442615 10647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:59.442646 10647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:52:59.444653 10647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:46507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=46505
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:59.444952 10647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:59.445199 10647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:59.451170 10657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.451159 10659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.451166 10656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.451449 10647 server_base.cc:1034] running on GCE node
I20260619 02:52:59.451749 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:59.452064 10647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:59.453227 10647 hybrid_clock.cc:648] HybridClock initialized: now 1781837579453209 us; error 37 us; skew 500 ppm
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:59.456990 10647 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:52:59.458102 10647 webserver.cc:460] Webserver started at http://127.26.98.2:46505/ using document root <none> and password file <none>
I20260619 02:52:59.458451 10647 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:59.458549 10647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:59.460877 10647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:59.462566 10666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.462925 10647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:59.463065 10647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "d37405c1845dc6abec1efccd98288901"
server_key_iv: "54ea529dbc88322afa1c4ca82ae4b375"
server_key_version: "encryptionkey@0"
I20260619 02:52:59.463763 10647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:59.485162 10647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:59.488859 10647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:59.489073 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:59.489436 10647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:59.489951 10647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:59.489995 10647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.490023 10647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:59.490034 10647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.499938 10647 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:46507
I20260619 02:52:59.500057 10779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:46507 every 8 connection(s)
I20260619 02:52:59.500474 10647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:59.501183 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10647
I20260619 02:52:59.501495 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:36497
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=35503
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:59.517138 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.502484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:53395 (local address 127.26.98.62:45487)
0619 02:52:59.502537 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.502539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.502562 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:52:59.503495 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.503497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.503518 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.503573 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:59.505518 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.506356 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.508299 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.508687 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.508824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.512429 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:59.512432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:59.512432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:59.512448 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:59.514499 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.515374 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.515376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.515377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.515493 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.516028 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.516030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.516030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.516275 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:59.516366 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.516818 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.516993 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:52:59.517401 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.501994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:53395)
0619 02:52:59.502384 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.502452 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.503328 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.503688 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.503694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.504701 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:59.505076 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.505087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.506614 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.506617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.508090 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.508100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.508879 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.508881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.509205 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.510009 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:59.510022 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:59.512222 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:59.514706 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.514721 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.514724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.515220 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.515679 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.515680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.515680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.515878 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.516395 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:59.516404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:59.516618 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.516745 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.516974 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260619 02:52:59.518172 10780 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:59.518311 10780 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:59.518646 10780 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
I20260619 02:52:59.519363 10454 ts_manager.cc:194] Registered new tserver with Master: 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507)
I20260619 02:52:59.519870 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:53395
W20260619 02:52:59.605386 10783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:59.605702 10783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:59.605746 10783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:59.607975 10783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:59.608060 10783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:59.608114 10783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:52:59.610183 10783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:36497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=35503
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:52:59.610554 10783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:59.610867 10783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:59.617882 10789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:59.617904 10791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.617997 10783 server_base.cc:1034] running on GCE node
W20260619 02:52:59.617895 10788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:59.618360 10783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:59.618708 10783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:59.619916 10783 hybrid_clock.cc:648] HybridClock initialized: now 1781837579619855 us; error 76 us; skew 500 ppm
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:59.623791 10783 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:52:59.624648 10783 webserver.cc:460] Webserver started at http://127.26.98.3:35503/ using document root <none> and password file <none>
I20260619 02:52:59.624949 10783 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:59.625039 10783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:59.626984 10783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:52:59.628075 10798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.628435 10783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:59.628562 10783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "0cb80e71069f4916a05359a44773ceef"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "89af67c55ab035ac73ac5b4353db8a16"
server_key_iv: "423ee106ae89ff4cea0f342c0e251cfe"
server_key_version: "encryptionkey@0"
I20260619 02:52:59.629098 10783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:59.641407 10783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:59.645160 10783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:59.645352 10783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:59.645761 10783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:59.646240 10783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:59.646281 10783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.646308 10783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:59.646332 10783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:59.654615 10783 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:36497
I20260619 02:52:59.654691 10911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:36497 every 8 connection(s)
I20260619 02:52:59.655241 10783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:59.660451 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10783
Jun 19 02:52:59 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837579, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:52:59.671205 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.661195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.26.98.62:45487)
0619 02:52:59.661335 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.661336 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.661712 (+ 376us) server_negotiation.cc:407] Connection header received
0619 02:52:59.661743 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.661744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.661758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.661832 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:59.663416 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.664212 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.665413 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.665712 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.665786 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.667373 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:59.667374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:59.667374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:59.667381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:59.669684 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.670141 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.670142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.670142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.670203 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.670474 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.670475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.670475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.670691 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:59.670942 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.670991 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.671088 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260619 02:52:59.671535 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.656533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:44125 (local address 127.26.98.62:45487)
0619 02:52:59.656618 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.656621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.656693 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:52:59.657583 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.657585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.657598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.657663 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:59.659712 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.660506 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.662993 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.663636 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.663743 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.667264 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:59.667267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:59.667267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:59.667277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:59.669404 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.670049 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.670051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.670051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.670162 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.670580 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.670581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.670581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.670690 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:59.670768 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.671318 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.671406 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:52:59.671701 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.656218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:44125)
0619 02:52:59.656515 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.656594 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.657401 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.657900 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.657905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.659229 (+ 1324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:59.659551 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.659560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.660745 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.660747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.662565 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.662572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.663835 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.663837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.664036 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.664959 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:59.664975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:59.667131 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:59.669520 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.669541 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.669545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.669941 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.670298 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:59.670300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:59.670301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:59.670505 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:59.670782 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:59.670789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:59.671049 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.671179 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.671415 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260619 02:52:59.672976 10912 heartbeater.cc:347] Connected to a master server at 127.26.98.62:45487
I20260619 02:52:59.673146 10912 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:59.673403 10912 heartbeater.cc:511] Master 127.26.98.62:45487 requested a full tablet report, sending...
I20260619 02:52:59.673877 10454 ts_manager.cc:194] Registered new tserver with Master: 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497)
I20260619 02:52:59.674292 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:44125
I20260619 02:52:59.683811 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:59.687695 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.684333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.26.98.62:45487)
0619 02:52:59.684422 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.684424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.684498 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:52:59.684582 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.684582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.684631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.684704 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:59.684993 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.685553 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.686563 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.686805 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.686980 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.687313 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:59.687422 (+ 109us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 02:52:59.687619 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40790: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 02:52:59.687769 10645 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40790: token expired
W20260619 02:52:59.688140 10382 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.98.62:45487)
I20260619 02:52:59.697945 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.688518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.26.98.62:45487)
0619 02:52:59.688597 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.688599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.688605 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:52:59.688624 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.688625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.688634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.688679 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:59.689224 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.689827 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.690736 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.691101 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.691199 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.692805 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:59.692808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:59.692808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:59.692822 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:59.695550 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.696302 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.696304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.696305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.696387 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:59.696815 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:59.696816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:59.696817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:59.697044 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:59.697154 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.697741 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.697852 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:52:59.700058 10454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:59.702260 10454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:59.705972 10576 tablet_service.cc:1467] Processing CreateTablet for tablet da20d71880994c379d0d5da6bdd3364f (DEFAULT_TABLE table=test-table [id=f7af6b32d21444deb34a9b6e207141b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:59.706291 10576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da20d71880994c379d0d5da6bdd3364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:59.708344 10534 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap starting.
I20260619 02:52:59.708844 10534 tablet_bootstrap.cc:654] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:59.709800 10534 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: No bootstrap required, opened a new log
I20260619 02:52:59.709880 10534 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:59.710049 10534 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.710155 10534 raft_consensus.cc:374] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:59.710235 10534 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3019050d32a14bfb83d03c5ee4ff6aab, State: Initialized, Role: FOLLOWER
I20260619 02:52:59.710307 10534 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.710417 10534 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:59.710592 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.705844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.26.98.3:36497)
0619 02:52:59.706166 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.706170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.706182 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:59.706234 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.706236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.706253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.706344 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:59.706600 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.707684 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.709307 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.710003 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.710106 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.710192 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.710431 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.710488 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260619 02:52:59.710577 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.705437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36497 (local address 127.0.0.1:59872)
0619 02:52:59.705793 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.705804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.705871 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.706456 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.706457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.706460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:59.706516 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.706517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.707790 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.707791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.709178 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.709181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.710245 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.710247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.710341 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.710348 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.710408 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.710478 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":147,"thread_start_us":65,"threads_started":1}
I20260619 02:52:59.710705 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.705436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.0.0.1:41644)
0619 02:52:59.705487 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.705503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.705547 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.706005 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.706006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.706008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:59.706088 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.706090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.707515 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.707516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.708913 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.708922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.710201 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.710203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.710286 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.710294 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.710375 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.710458 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":10}
I20260619 02:52:59.711712 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.705566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.26.98.2:46507)
0619 02:52:59.705780 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.705787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.705804 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:52:59.705863 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.705866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.705886 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.706006 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:59.706168 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.707403 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.709119 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.710004 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.710158 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.710218 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.711534 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.711606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":118,"threads_started":1}
I20260619 02:52:59.711663 10846 tablet_service.cc:1467] Processing CreateTablet for tablet da20d71880994c379d0d5da6bdd3364f (DEFAULT_TABLE table=test-table [id=f7af6b32d21444deb34a9b6e207141b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:59.712152 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da20d71880994c379d0d5da6bdd3364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:59.712713 10714 tablet_service.cc:1467] Processing CreateTablet for tablet da20d71880994c379d0d5da6bdd3364f (DEFAULT_TABLE table=test-table [id=f7af6b32d21444deb34a9b6e207141b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:59.713126 10714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da20d71880994c379d0d5da6bdd3364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:59.716239 10925 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Bootstrap starting.
I20260619 02:52:59.717226 10925 tablet_bootstrap.cc:654] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:59.717552 10926 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Bootstrap starting.
I20260619 02:52:59.717984 10925 log.cc:824] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:59.718520 10926 tablet_bootstrap.cc:654] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:59.719205 10925 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: No bootstrap required, opened a new log
I20260619 02:52:59.719296 10925 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:59.719509 10926 log.cc:824] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:59.720724 10926 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: No bootstrap required, opened a new log
I20260619 02:52:59.720821 10926 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:52:59.721534 10925 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.721761 10925 raft_consensus.cc:374] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:59.721804 10925 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb80e71069f4916a05359a44773ceef, State: Initialized, Role: FOLLOWER
I20260619 02:52:59.721976 10925 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.722400 10912 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
I20260619 02:52:59.722400 10925 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:52:59.723325 10926 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.723599 10926 raft_consensus.cc:374] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:59.723629 10926 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885bbfb8247d4d04b7c0a07885a7a3f6, State: Initialized, Role: FOLLOWER
I20260619 02:52:59.723774 10926 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.724159 10926 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260619 02:52:59.724241 10780 heartbeater.cc:503] Master 127.26.98.62:45487 was elected leader, sending a full tablet report...
W20260619 02:52:59.751295 10781 tablet.cc:2369] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:59.812264 10643 tablet.cc:2369] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:59.842283 10929 raft_consensus.cc:482] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:59.842459 10929 raft_consensus.cc:504] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.842933 10929 leader_election.cc:288] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507), 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
I20260619 02:52:59.848141 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.843289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:43117 (local address 127.26.98.2:46507)
0619 02:52:59.843337 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.843339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.843525 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:52:59.843561 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.843562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.843581 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.843650 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:59.843892 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.844737 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.846622 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.847474 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.847601 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.847618 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.848000 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.848058 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:52:59.848284 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.843213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.26.98.3:43895)
0619 02:52:59.843296 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.843310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.843379 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.843666 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.843667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.843669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:59.843794 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.843797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.844899 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.844901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.846427 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.846432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.847763 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.847764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.847872 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.847880 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.847952 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.848021 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":129}
I20260619 02:52:59.848296 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.843213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.26.98.3:43117)
0619 02:52:59.843433 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.843442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:59.843509 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:59.843666 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:59.843667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:59.843669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:59.843826 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.843828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.844899 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.844901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.846426 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:59.846432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.847763 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.847764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:59.847872 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.847884 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:52:59.847952 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:52:59.848018 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":112,"thread_start_us":115,"threads_started":1}
I20260619 02:52:59.848336 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.843284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:43895 (local address 127.26.98.1:36235)
0619 02:52:59.843445 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.843447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.843458 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:59.843518 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.843519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.843534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.843646 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:59.843881 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.844713 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.846622 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.847479 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.847602 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.847618 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.848207 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.848264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260619 02:52:59.848716 10596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "0cb80e71069f4916a05359a44773ceef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" is_pre_election: true
I20260619 02:52:59.848778 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "0cb80e71069f4916a05359a44773ceef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" is_pre_election: true
I20260619 02:52:59.848930 10734 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb80e71069f4916a05359a44773ceef in term 0.
I20260619 02:52:59.848930 10596 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb80e71069f4916a05359a44773ceef in term 0.
I20260619 02:52:59.849295 10799 leader_election.cc:302] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb80e71069f4916a05359a44773ceef, 3019050d32a14bfb83d03c5ee4ff6aab; no voters:
I20260619 02:52:59.849582 10929 raft_consensus.cc:2781] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:59.849709 10929 raft_consensus.cc:482] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:59.849735 10929 raft_consensus.cc:3037] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:59.850637 10929 raft_consensus.cc:504] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.850826 10929 leader_election.cc:288] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [CANDIDATE]: Term 1 election: Requested vote from peers 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507), 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
I20260619 02:52:59.851131 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "0cb80e71069f4916a05359a44773ceef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
I20260619 02:52:59.851158 10596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "0cb80e71069f4916a05359a44773ceef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
I20260619 02:52:59.851243 10734 raft_consensus.cc:3037] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:59.851284 10596 raft_consensus.cc:3037] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:59.852088 10734 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb80e71069f4916a05359a44773ceef in term 1.
I20260619 02:52:59.852252 10596 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb80e71069f4916a05359a44773ceef in term 1.
I20260619 02:52:59.852491 10801 leader_election.cc:302] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb80e71069f4916a05359a44773ceef, 885bbfb8247d4d04b7c0a07885a7a3f6; no voters:
I20260619 02:52:59.852646 10929 raft_consensus.cc:2781] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:59.852873 10929 raft_consensus.cc:686] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 1 LEADER]: Becoming Leader. State: Replica: 0cb80e71069f4916a05359a44773ceef, State: Running, Role: LEADER
I20260619 02:52:59.852993 10929 consensus_queue.cc:237] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:52:59.854228 10454 catalog_manager.cc:5495] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb80e71069f4916a05359a44773ceef (127.26.98.3). New cstate: current_term: 1 leader_uuid: "0cb80e71069f4916a05359a44773ceef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } health_report { overall_health: UNKNOWN } } }
I20260619 02:52:59.898814 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.895264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.26.98.3:36497)
0619 02:52:59.895388 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:59.895390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:59.895402 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:59.895455 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:59.895456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:59.895486 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:59.895569 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:59.895880 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.896707 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.897808 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:59.898046 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:59.898129 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:59.898357 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:59.898508 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:59.898594 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:59.898611 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 02:52:59.898640 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 02:52:59.902114 10734 raft_consensus.cc:1261] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Refusing update from remote peer 0cb80e71069f4916a05359a44773ceef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:59.902114 10596 raft_consensus.cc:1261] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Refusing update from remote peer 0cb80e71069f4916a05359a44773ceef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:59.902840 10929 consensus_queue.cc:1035] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [LEADER]: Connected to new peer: Peer: permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:59.903116 10933 consensus_queue.cc:1035] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:52:59.905857 10913 tablet.cc:2369] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:59.906809 10938 mvcc.cc:205] Tried to move back new op lower bound from 7298406727273250816 to 7298406727078690816. Current Snapshot: MvccSnapshot[applied={T|T < 7298406727273250816}]
I20260619 02:52:59.906981 10937 mvcc.cc:205] Tried to move back new op lower bound from 7298406727273250816 to 7298406727078690816. Current Snapshot: MvccSnapshot[applied={T|T < 7298406727273250816 or (T in {7298406727273250816})}]
I20260619 02:52:59.911401 10453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:59.911559 10453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:59.912698 10453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Sending DeleteTablet for 3 replicas of tablet da20d71880994c379d0d5da6bdd3364f
I20260619 02:52:59.913228 10714 tablet_service.cc:1514] Processing DeleteTablet for tablet da20d71880994c379d0d5da6bdd3364f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:59 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:41644
I20260619 02:52:59.913385 10576 tablet_service.cc:1514] Processing DeleteTablet for tablet da20d71880994c379d0d5da6bdd3364f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:59 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45932
I20260619 02:52:59.913395 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10492
I20260619 02:52:59.913662 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet da20d71880994c379d0d5da6bdd3364f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:59 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:59872
I20260619 02:52:59.914443 10945 tablet_replica.cc:330] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: stopping tablet replica
I20260619 02:52:59.914662 10945 raft_consensus.cc:2229] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:59.914847 10945 raft_consensus.cc:2258] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:59.919472 10945 ts_tablet_manager.cc:1905] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:52:59.919636 10438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:36235 (error 108)
I20260619 02:52:59.920266 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10647
I20260619 02:52:59.920362 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.920211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.0.0.1:45942)
0619 02:52:59.920286 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.920324 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:52:59.921319 10438 catalog_manager.cc:4584] TS 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da20d71880994c379d0d5da6bdd3364f: Network error: Client connection negotiation failed: client connection to 127.26.98.1:36235: connect: Connection refused (error 111)
I20260619 02:52:59.922438 10946 tablet_replica.cc:330] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: stopping tablet replica
I20260619 02:52:59.922767 10946 raft_consensus.cc:2229] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:52:59.923017 10946 raft_consensus.cc:2258] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:59.923596 10946 ts_tablet_manager.cc:1905] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:59.925540 10946 ts_tablet_manager.cc:1918] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:59.925709 10946 log.cc:1196] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/da20d71880994c379d0d5da6bdd3364f
I20260619 02:52:59.926017 10946 ts_tablet_manager.cc:1939] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Deleting consensus metadata
I20260619 02:52:59.926720 10441 catalog_manager.cc:4841] TS 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497): tablet da20d71880994c379d0d5da6bdd3364f (table test-table [id=f7af6b32d21444deb34a9b6e207141b2]) successfully deleted
I20260619 02:52:59.927007 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:59.926768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.0.0.1:41656)
0619 02:52:59.926912 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:59.926961 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.2:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 02:52:59.927062 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10783
I20260619 02:52:59.932562 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 10421
I20260619 02:53:19.938761 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--webserver_interface=127.26.98.62
--webserver_port=35839
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:20.042829 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:20.043180 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:20.043224 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:20.044694 10948 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:53:20.044764 10948 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:53:20.044781 10948 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:53:20.044797 10948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:20.044814 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:20.044833 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:20.044857 10948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:20.044878 10948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:20.046741 10948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:20.047099 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:20.047374 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:20.053146 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.053184 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.053254 10948 server_base.cc:1034] running on GCE node
W20260619 02:53:20.053146 10956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.053785 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:20.054095 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:20.055227 10948 hybrid_clock.cc:648] HybridClock initialized: now 1781837600055217 us; error 31 us; skew 500 ppm
Jun 19 02:53:20 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:20.057991 10948 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:53:20.058516 10948 webserver.cc:460] Webserver started at http://127.26.98.62:35839/ using document root <none> and password file <none>
I20260619 02:53:20.058754 10948 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:20.058806 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:20.060381 10948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.061266 10963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.061489 10948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.061586 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "338a0248b1c846449ab56cdd9d855328"
format_stamp: "Formatted at 2026-06-19 02:52:37 on dist-test-slave-0f9d"
server_key: "32b75c931f0201a22f1f0d737cb145a7"
server_key_iv: "29b7764cbff554e790eb28b3f314d60a"
server_key_version: "encryptionkey@0"
I20260619 02:53:20.062059 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:20.076215 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:20.079145 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:20.079321 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:20.084393 10948 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45487
I20260619 02:53:20.084434 11015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45487 every 8 connection(s)
I20260619 02:53:20.084940 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:20.085932 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 10948
I20260619 02:53:20.086246 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:36235
--local_ip_for_outbound_sockets=127.26.98.1
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=46745
--webserver_interface=127.26.98.1
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:20.087131 11016 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:53:20.087927 11016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap starting.
I20260619 02:53:20.101696 11016 log.cc:824] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:20.106226 11016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:20.106792 11016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Bootstrap complete.
I20260619 02:53:20.112325 11016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:53:20.112854 11016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Initialized, Role: FOLLOWER
I20260619 02:53:20.113169 11016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:53:20.113271 11016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:20.113332 11016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:20.113382 11016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:53:20.115463 11016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:53:20.115656 11016 leader_election.cc:302] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338a0248b1c846449ab56cdd9d855328; no voters:
I20260619 02:53:20.115911 11016 leader_election.cc:288] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:53:20.116037 11020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:53:20.116235 11020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [term 3 LEADER]: Becoming Leader. State: Replica: 338a0248b1c846449ab56cdd9d855328, State: Running, Role: LEADER
I20260619 02:53:20.116367 11020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } }
I20260619 02:53:20.116559 11016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:20.116735 11021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:53:20.116827 11021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:20.117019 11022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338a0248b1c846449ab56cdd9d855328. Latest consensus state: current_term: 3 leader_uuid: "338a0248b1c846449ab56cdd9d855328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338a0248b1c846449ab56cdd9d855328" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45487 } } }
I20260619 02:53:20.117076 11022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:20.117413 11024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:20.118502 11024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:20.118640 11024 catalog_manager.cc:1259] Loaded cluster ID: 2db7a836b75e4d359411617945888f74
I20260619 02:53:20.118680 11024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:20.120276 11024 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:20.120522 11024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Loaded TSK: 0
I20260619 02:53:20.120569 11024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 338a0248b1c846449ab56cdd9d855328: Loaded TSK: 1
F20260619 02:53:20.120865 11024 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faa611da9ad google::LogMessage::Fail() at ??:0
@ 0x7faa611dd88a google::LogMessage::SendToLog() at ??:0
@ 0x7faa611da4de google::LogMessage::Flush() at ??:0
@ 0x7faa611dba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faa63947104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faa63944d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faa691341cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faa6913448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faa69134594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faa690fac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faa69104ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faa630a80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faa6309d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faa63f016db start_thread at ??:0
@ 0x7faa5fcd861f clone at ??:0
W20260619 02:53:20.219596 11018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:20.219877 11018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:20.219920 11018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:20.221858 11018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:20.221951 11018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:20.222007 11018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:20.224632 11018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:36235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=46745
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:20.224996 11018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:20.225292 11018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:20.231253 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.231364 11018 server_base.cc:1034] running on GCE node
W20260619 02:53:20.231523 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.231288 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.231825 11018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:20.232157 11018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:20.233358 11018 hybrid_clock.cc:648] HybridClock initialized: now 1781837600233340 us; error 39 us; skew 500 ppm
Jun 19 02:53:20 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:20.236650 11018 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:53:20.237308 11018 webserver.cc:460] Webserver started at http://127.26.98.1:46745/ using document root <none> and password file <none>
I20260619 02:53:20.237608 11018 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:20.237672 11018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:20.239563 11018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:20.240854 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.241225 11018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.241370 11018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "32eded6e0647efa8ee412e9887289b6f"
server_key_iv: "a91e5f4e06b5234ee4546c3673cf4ee6"
server_key_version: "encryptionkey@0"
I20260619 02:53:20.241950 11018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:20.262337 11018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:20.265269 11018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:20.265451 11018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:20.265730 11018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:20.266777 11074 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:53:20.269879 11018 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:53:20.269984 11018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 02:53:20.270037 11018 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:53:20.271003 11018 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:53:20.271060 11018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.271199 11074 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap starting.
I20260619 02:53:20.280160 11018 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:36235
I20260619 02:53:20.280362 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:36235 every 8 connection(s)
I20260619 02:53:20.280695 11018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:20.284691 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11018
I20260619 02:53:20.285038 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:46507
--local_ip_for_outbound_sockets=127.26.98.2
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=46505
--webserver_interface=127.26.98.2
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:20.287439 11074 log.cc:824] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:20.291857 11074 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:20.292361 11074 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Bootstrap complete.
I20260619 02:53:20.293633 11074 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.008s
I20260619 02:53:20.295825 11074 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:20.296308 11074 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3019050d32a14bfb83d03c5ee4ff6aab, State: Initialized, Role: FOLLOWER
I20260619 02:53:20.296550 11074 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:20.296985 11074 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260619 02:53:20.389516 11185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:20.389894 11185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:20.389948 11185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:20.390200 11184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:20.281620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:34111)
0619 02:53:20.281875 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.281927 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:20.282519 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:20.389933 (+107414us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
W20260619 02:53:20.390540 11182 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:45487 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:45487: Client connection negotiation failed: client connection to 127.26.98.62:45487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:53:20.391000 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.390830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:46135)
0619 02:53:20.390909 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.390968 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 02:53:20.391692 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.391481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:52459)
0619 02:53:20.391580 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.391642 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 02:53:20.392185 11182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:53:20.392236 11185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:20.392331 11185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:20.392397 11185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:20.395962 11185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:46507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=46505
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:20.396385 11185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:20.396695 11185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:20.404597 11193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.404644 11195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.404597 11192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.406088 11185 server_base.cc:1034] running on GCE node
I20260619 02:53:20.406463 11185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:20.407013 11185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:20.408242 11185 hybrid_clock.cc:648] HybridClock initialized: now 1781837600408197 us; error 72 us; skew 500 ppm
Jun 19 02:53:20 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:20.411583 11185 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:53:20.412216 11185 webserver.cc:460] Webserver started at http://127.26.98.2:46505/ using document root <none> and password file <none>
I20260619 02:53:20.412500 11185 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:20.412559 11185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:20.414342 11185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:53:20.415572 11202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.415874 11185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:53:20.415992 11185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "885bbfb8247d4d04b7c0a07885a7a3f6"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "d37405c1845dc6abec1efccd98288901"
server_key_iv: "54ea529dbc88322afa1c4ca82ae4b375"
server_key_version: "encryptionkey@0"
I20260619 02:53:20.416769 11185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:20.431046 11185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:20.434309 11185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:20.434535 11185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:20.434811 11185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:20.435371 11209 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:53:20.437296 11185 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:53:20.437390 11185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:53:20.437440 11185 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:53:20.438421 11185 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:53:20.438473 11185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:53:20.438587 11209 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Bootstrap starting.
I20260619 02:53:20.440882 11209 log.cc:824] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:20.444079 11209 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:20.444429 11209 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Bootstrap complete.
I20260619 02:53:20.444592 11209 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:53:20.446741 11209 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:20.447136 11209 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885bbfb8247d4d04b7c0a07885a7a3f6, State: Initialized, Role: FOLLOWER
I20260619 02:53:20.447363 11209 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:20.447505 11185 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:46507
I20260619 02:53:20.447676 11209 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260619 02:53:20.447830 11317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:46507 every 8 connection(s)
I20260619 02:53:20.448001 11185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:20.449191 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.448740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:38663)
0619 02:53:20.448950 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.449042 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
W20260619 02:53:20.449441 11319 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:45487 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:45487: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
I20260619 02:53:20.449900 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.449730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:35189)
0619 02:53:20.449836 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.449872 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 02:53:20.450268 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.450162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:41539)
0619 02:53:20.450218 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.450246 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 02:53:20.450371 11319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:53:20.452895 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11185
I20260619 02:53:20.453204 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:36497
--local_ip_for_outbound_sockets=127.26.98.3
--tserver_master_addrs=127.26.98.62:45487
--webserver_port=35503
--webserver_interface=127.26.98.3
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:20.554034 11322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:20.554394 11322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:20.554428 11322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:20.556373 11322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:20.556493 11322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:20.556535 11322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:20.558844 11322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:36497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=35503
--tserver_master_addrs=127.26.98.62:45487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:20.559259 11322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:20.559559 11322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:20.566385 11330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.566447 11328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:20.566504 11327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:20.566936 11322 server_base.cc:1034] running on GCE node
I20260619 02:53:20.567211 11322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:20.567544 11322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:20.568830 11322 hybrid_clock.cc:648] HybridClock initialized: now 1781837600568810 us; error 39 us; skew 500 ppm
Jun 19 02:53:20 dist-test-slave-0f9d krb5kdc[9873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:20.573011 11322 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:53:20.573875 11322 webserver.cc:460] Webserver started at http://127.26.98.3:35503/ using document root <none> and password file <none>
I20260619 02:53:20.574188 11322 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:20.574259 11322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:20.576350 11322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.577323 11337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.577611 11322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:20.577730 11322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "0cb80e71069f4916a05359a44773ceef"
format_stamp: "Formatted at 2026-06-19 02:52:38 on dist-test-slave-0f9d"
server_key: "89af67c55ab035ac73ac5b4353db8a16"
server_key_iv: "423ee106ae89ff4cea0f342c0e251cfe"
server_key_version: "encryptionkey@0"
I20260619 02:53:20.578236 11322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:20.604902 11322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:20.608368 11322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:20.608582 11322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:20.608898 11322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:20.609297 11322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:20.609335 11322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.609380 11322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:20.609406 11322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:20.616812 11322 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:36497
I20260619 02:53:20.616909 11450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:36497 every 8 connection(s)
I20260619 02:53:20.617305 11322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:20.618582 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.618132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:57075)
0619 02:53:20.618324 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.618409 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
W20260619 02:53:20.619060 11451 heartbeater.cc:644] Failed to heartbeat to 127.26.98.62:45487 (0 consecutive failures): Network error: Failed to ping master at 127.26.98.62:45487: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
I20260619 02:53:20.619393 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.619299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:42955)
0619 02:53:20.619344 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.619372 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:53:20.619656 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.619568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:57111)
0619 02:53:20.619601 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.619622 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 02:53:20.619748 11451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:53:20.621013 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11322
I20260619 02:53:21.392974 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.392644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:36357)
0619 02:53:21.392857 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.392908 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260619 02:53:21.451086 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.450781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:35819)
0619 02:53:21.450966 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.451019 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260619 02:53:21.620550 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.620208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:45283)
0619 02:53:21.620421 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.620473 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260619 02:53:21.887559 11458 raft_consensus.cc:482] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:21.887743 11458 raft_consensus.cc:504] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:21.888195 11458 leader_election.cc:288] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497), 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
Jun 19 02:53:21 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.1@KRBTEST.COM
Jun 19 02:53:21 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.3@KRBTEST.COM
I20260619 02:53:21.902298 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.888615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:47013 (local address 127.26.98.3:36497)
0619 02:53:21.889027 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:21.889037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:21.889077 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:53:21.889961 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:21.889965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:21.890032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:21.890204 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:21.890743 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.892351 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894001 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.894567 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894710 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:21.898261 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:21.898267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:21.898267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:21.898282 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:21.900451 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:21.900937 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:21.900941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:21.900942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:21.901048 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:21.901307 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:21.901308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:21.901309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:21.901616 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:21.901700 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:21.901985 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:53:21.902145 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":95,"threads_started":1}
I20260619 02:53:21.902302 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.888613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:49423 (local address 127.26.98.1:36235)
0619 02:53:21.888829 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:21.888836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:21.888853 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:53:21.888975 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:21.888978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:21.889177 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:21.890328 (+ 1151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:21.890770 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.892367 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894019 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.894578 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894710 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:21.898068 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:21.898083 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:21.898084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:21.898103 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:21.900339 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:21.900937 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:21.900941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:21.900942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:21.901048 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:21.901298 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:21.901299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:21.901299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:21.901696 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:21.901822 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:21.902026 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:53:21.902185 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260619 02:53:21.902295 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.888454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:36497 (local address 127.26.98.2:47013)
0619 02:53:21.888813 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.888825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:21.888907 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:21.890209 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:21.890210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:21.890466 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:21.890617 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:21.890626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.892523 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.892524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:21.893873 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:21.893879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.894712 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:21.894994 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:21.895712 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:21.895717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:21.898084 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:21.900558 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:21.900559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:21.900560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:21.900847 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:21.901117 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:21.901118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:21.901118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:21.901240 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:21.901683 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:21.901691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:21.901832 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:53:21.901926 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:53:21.902159 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":189,"thread_start_us":87,"threads_started":1}
I20260619 02:53:21.902390 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.888454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:36235 (local address 127.26.98.2:49423)
0619 02:53:21.888503 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.888539 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:21.888747 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:21.889359 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:21.889366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:21.889857 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:21.890617 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:21.890627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.892507 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.892509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:21.893873 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:21.893878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:21.894706 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:21.894708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:21.894922 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:21.895654 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:21.895670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:21.897874 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:21.900457 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:21.900473 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:21.900476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:21.900846 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:21.901121 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:21.901122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:21.901123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:21.901240 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:21.901816 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:21.901818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:21.901875 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:53:21.901950 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:53:21.902242 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":196}
I20260619 02:53:21.903602 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" is_pre_election: true
I20260619 02:53:21.903807 11136 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885bbfb8247d4d04b7c0a07885a7a3f6 in term 1.
W20260619 02:53:21.904090 11206 leader_election.cc:341] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497): Not found: Tablet not found: da20d71880994c379d0d5da6bdd3364f
I20260619 02:53:21.904186 11206 leader_election.cc:302] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3019050d32a14bfb83d03c5ee4ff6aab, 885bbfb8247d4d04b7c0a07885a7a3f6; no voters: 0cb80e71069f4916a05359a44773ceef
I20260619 02:53:21.904322 11458 raft_consensus.cc:2781] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:53:21.904390 11458 raft_consensus.cc:482] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:21.904407 11458 raft_consensus.cc:3037] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:53:21.905974 11458 raft_consensus.cc:504] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:21.906152 11458 leader_election.cc:288] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 election: Requested vote from peers 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497), 3019050d32a14bfb83d03c5ee4ff6aab (127.26.98.1:36235)
W20260619 02:53:21.906540 11206 leader_election.cc:341] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497): Not found: Tablet not found: da20d71880994c379d0d5da6bdd3364f
I20260619 02:53:21.906509 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da20d71880994c379d0d5da6bdd3364f" candidate_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3019050d32a14bfb83d03c5ee4ff6aab"
I20260619 02:53:21.906628 11136 raft_consensus.cc:3037] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:53:21.908238 11136 raft_consensus.cc:2454] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885bbfb8247d4d04b7c0a07885a7a3f6 in term 2.
I20260619 02:53:21.908514 11203 leader_election.cc:302] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3019050d32a14bfb83d03c5ee4ff6aab, 885bbfb8247d4d04b7c0a07885a7a3f6; no voters: 0cb80e71069f4916a05359a44773ceef
I20260619 02:53:21.908625 11458 raft_consensus.cc:2781] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:53:21.908803 11458 raft_consensus.cc:686] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [term 2 LEADER]: Becoming Leader. State: Replica: 885bbfb8247d4d04b7c0a07885a7a3f6, State: Running, Role: LEADER
I20260619 02:53:21.908921 11458 consensus_queue.cc:237] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:21.909211 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.909073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:37809)
0619 02:53:21.909123 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.909173 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:53:22.353674 11206 consensus_peers.cc:487] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 -> Peer 0cb80e71069f4916a05359a44773ceef (127.26.98.3:36497): Couldn't send request to peer 0cb80e71069f4916a05359a44773ceef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da20d71880994c379d0d5da6bdd3364f. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:53:22.378823 11136 raft_consensus.cc:1261] T da20d71880994c379d0d5da6bdd3364f P 3019050d32a14bfb83d03c5ee4ff6aab [term 2 FOLLOWER]: Refusing update from remote peer 885bbfb8247d4d04b7c0a07885a7a3f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 02:53:22.379401 11458 consensus_queue.cc:1035] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 02:53:22.381343 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.380886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:39429)
0619 02:53:22.381084 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.381161 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
I20260619 02:53:22.382184 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.382032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:53489)
0619 02:53:22.382084 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.382135 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:53:22.382285 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.382171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:36711)
0619 02:53:22.382218 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.382249 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:53:22.621548 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.621233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:50215)
0619 02:53:22.621427 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.621481 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260619 02:53:22.929387 11470 ts_tablet_manager.cc:927] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Initiating tablet copy from peer 885bbfb8247d4d04b7c0a07885a7a3f6 (127.26.98.2:46507)
I20260619 02:53:22.929709 11470 tablet_copy_client.cc:316] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: tablet copy: Beginning tablet copy session from remote peer at address 127.26.98.2:46507
Jun 19 02:53:22 dist-test-slave-0f9d krb5kdc[9873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837600, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.2@KRBTEST.COM
I20260619 02:53:22.944837 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.930319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:42029 (local address 127.26.98.2:46507)
0619 02:53:22.930534 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:22.930539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:22.930551 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:22.930583 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:22.930586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:22.930603 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:22.930706 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:22.931426 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:22.932298 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:22.933366 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:22.933769 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:22.933847 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:22.941423 (+ 7576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:22.941439 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:22.941440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:22.941451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:22.943380 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:22.943853 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:22.943858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:22.943859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:22.943928 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:22.944115 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:22.944116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:22.944117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:22.944307 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:22.944393 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:22.944603 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:53:22.944723 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260619 02:53:22.944860 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.930130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:46507 (local address 127.26.98.3:42029)
0619 02:53:22.930227 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.930266 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:22.930382 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:22.930782 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:22.930789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:22.931201 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:22.931308 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:22.931311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:22.932438 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:22.932440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:22.933232 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:22.933237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:22.933970 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:22.933972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:22.934105 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:22.939425 (+ 5320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:22.939433 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:22.941271 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:22.943483 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:22.943500 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:22.943503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:22.943761 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:22.943992 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:22.943993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:22.943994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:22.944062 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:22.944398 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:22.944408 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:22.944479 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:53:22.944558 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:53:22.944713 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:53:22.945426 11292 tablet_copy_service.cc:140] P 885bbfb8247d4d04b7c0a07885a7a3f6: Received BeginTabletCopySession request for tablet da20d71880994c379d0d5da6bdd3364f from peer 0cb80e71069f4916a05359a44773ceef ({username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:42029)
I20260619 02:53:22.945508 11292 tablet_copy_service.cc:161] P 885bbfb8247d4d04b7c0a07885a7a3f6: Beginning new tablet copy session on tablet da20d71880994c379d0d5da6bdd3364f from peer 0cb80e71069f4916a05359a44773ceef at {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:42029: session id = 0cb80e71069f4916a05359a44773ceef-da20d71880994c379d0d5da6bdd3364f
I20260619 02:53:22.946249 11292 tablet_copy_source_session.cc:205] T da20d71880994c379d0d5da6bdd3364f P 885bbfb8247d4d04b7c0a07885a7a3f6: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:53:22.948192 11470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da20d71880994c379d0d5da6bdd3364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:22.952555 11470 tablet_copy_client.cc:791] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: tablet copy: Starting download of 0 data blocks...
I20260619 02:53:22.952749 11470 tablet_copy_client.cc:655] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: tablet copy: Starting download of 1 WAL segments...
I20260619 02:53:22.954270 11470 tablet_copy_client.cc:531] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:53:22.955360 11470 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Bootstrap starting.
I20260619 02:53:22.957206 11470 log.cc:824] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:22.960909 11470 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:22.961221 11470 tablet_bootstrap.cc:492] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Bootstrap complete.
I20260619 02:53:22.961396 11470 ts_tablet_manager.cc:1397] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:53:22.962625 11470 raft_consensus.cc:348] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:22.962860 11470 raft_consensus.cc:729] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb80e71069f4916a05359a44773ceef, State: Initialized, Role: FOLLOWER
I20260619 02:53:22.963061 11470 consensus_queue.cc:260] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb80e71069f4916a05359a44773ceef" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 36497 } } peers { permanent_uuid: "885bbfb8247d4d04b7c0a07885a7a3f6" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 46507 } } peers { permanent_uuid: "3019050d32a14bfb83d03c5ee4ff6aab" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 36235 } }
I20260619 02:53:22.963398 11470 ts_tablet_manager.cc:1428] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260619 02:53:22.963799 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.963657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:60683)
0619 02:53:22.963712 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.963755 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:53:22.963935 11292 tablet_copy_service.cc:342] P 885bbfb8247d4d04b7c0a07885a7a3f6: Request end of tablet copy session 0cb80e71069f4916a05359a44773ceef-da20d71880994c379d0d5da6bdd3364f received from {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:42029
I20260619 02:53:22.964012 11292 tablet_copy_service.cc:433] P 885bbfb8247d4d04b7c0a07885a7a3f6: ending tablet copy session 0cb80e71069f4916a05359a44773ceef-da20d71880994c379d0d5da6bdd3364f on tablet da20d71880994c379d0d5da6bdd3364f with peer 0cb80e71069f4916a05359a44773ceef
I20260619 02:53:22.964165 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.964091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:42583)
0619 02:53:22.964121 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.964148 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:53:23.383116 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.382787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:40917)
0619 02:53:23.382961 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.383019 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260619 02:53:23.383186 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.382928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:59599)
0619 02:53:23.383086 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.383133 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260619 02:53:23.509758 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.509586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:40891)
0619 02:53:23.509658 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.509702 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:53:23.509802 11405 raft_consensus.cc:1203] T da20d71880994c379d0d5da6bdd3364f P 0cb80e71069f4916a05359a44773ceef [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 02:53:23.510514 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.510264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:53023)
0619 02:53:23.510390 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.510442 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":47,"threads_started":1}
I20260619 02:53:24.384208 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.383883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:33959)
0619 02:53:24.384070 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.384141 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260619 02:53:24.511018 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.510593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:43917)
0619 02:53:24.510818 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.510936 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
I20260619 02:53:24.511550 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.511077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:33997)
0619 02:53:24.511341 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.511496 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260619 02:53:25.385226 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.384938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:44857)
0619 02:53:25.385122 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.385156 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260619 02:53:25.512079 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.511790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:51731)
0619 02:53:25.511964 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.512009 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260619 02:53:25.512383 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.512124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:35949)
0619 02:53:25.512281 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.512333 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260619 02:53:26.386632 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:26.386286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:49183)
0619 02:53:26.386508 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:26.386566 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
I20260619 02:53:26.513298 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:26.513040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:49655)
0619 02:53:26.513196 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:26.513243 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260619 02:53:26.513278 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:26.512854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:55051)
0619 02:53:26.513152 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:26.513210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260619 02:53:27.387876 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.387438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:43201)
0619 02:53:27.387746 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.387811 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260619 02:53:27.514374 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.514087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:38435)
0619 02:53:27.514276 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.514315 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260619 02:53:27.514374 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.514087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:55407)
0619 02:53:27.514268 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.514310 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260619 02:53:28.389060 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.388702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:44193)
0619 02:53:28.388927 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.388987 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260619 02:53:28.515261 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.514995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:56687)
0619 02:53:28.515149 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.515198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260619 02:53:28.515260 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.514995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:40643)
0619 02:53:28.515161 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.515199 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260619 02:53:29.390101 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:29.389788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:52207)
0619 02:53:29.389980 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:29.390037 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260619 02:53:29.516141 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:29.515866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:34543)
0619 02:53:29.516020 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:29.516078 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260619 02:53:29.516141 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:29.515866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:58133)
0619 02:53:29.516041 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:29.516083 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260619 02:53:30.391428 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.390820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:45127)
0619 02:53:30.391175 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.391258 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260619 02:53:30.517068 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.516759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:37255)
0619 02:53:30.516966 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.517003 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260619 02:53:30.517143 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.516759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:57169)
0619 02:53:30.516978 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.517011 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260619 02:53:31.392645 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.392327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:59057)
0619 02:53:31.392523 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.392581 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260619 02:53:31.518023 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.517723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:55351)
0619 02:53:31.517913 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.517949 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260619 02:53:31.518014 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.517721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:52257)
0619 02:53:31.517904 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.517955 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260619 02:53:32.393823 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:32.393423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:57711)
0619 02:53:32.393684 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:32.393750 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260619 02:53:32.519044 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:32.518720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:43467)
0619 02:53:32.518937 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:32.518981 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260619 02:53:32.519045 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:32.518722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:60573)
0619 02:53:32.518929 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:32.518982 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260619 02:53:33.394737 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:33.394450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:59779)
0619 02:53:33.394614 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:33.394669 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260619 02:53:33.520015 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:33.519727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:43905)
0619 02:53:33.519904 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:33.519950 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260619 02:53:33.520013 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:33.519739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:42975)
0619 02:53:33.519903 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:33.519950 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260619 02:53:34.395606 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:34.395309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:37279)
0619 02:53:34.395476 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:34.395538 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260619 02:53:34.521008 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:34.520732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:47755)
0619 02:53:34.520897 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:34.520945 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260619 02:53:34.521015 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:34.520732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:53595)
0619 02:53:34.520897 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:34.520945 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260619 02:53:35.396754 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:35.396445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.1:45799)
0619 02:53:35.396625 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:35.396680 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260619 02:53:35.522006 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:35.521712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.3:53337)
0619 02:53:35.521890 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:35.521934 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260619 02:53:35.522006 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:35.521713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:45487 (local address 127.26.98.2:36807)
0619 02:53:35.521884 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:35.521932 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.62:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260619 02:53:35.631234 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11018
I20260619 02:53:35.637001 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11185
I20260619 02:53:35.642486 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11322
2026-06-19T02:53:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59044 ms)
[----------] 30 tests from SecurityITest (263011 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:35 dist-test-slave-0f9d krb5kdc[11519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:35 dist-test-slave-0f9d krb5kdc[11519](info): set up 2 sockets
Jun 19 02:53:35 dist-test-slave-0f9d krb5kdc[11519](info): commencing operation
krb5kdc: starting...
W20260619 02:53:36.659018 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:36 dist-test-slave-0f9d krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:53:36.798730 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44963
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:36.896778 11532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:36.897121 11532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:36.897153 11532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:36.898869 11532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:36.899004 11532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:36.899019 11532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:36.899034 11532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:36.899045 11532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:36.901233 11532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:36.901607 11532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:36.901887 11532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:36.908893 11540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:36.908893 11538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:36.908938 11532 server_base.cc:1034] running on GCE node
W20260619 02:53:36.908893 11537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:36.909480 11532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:36.909902 11532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:36.911124 11532 hybrid_clock.cc:648] HybridClock initialized: now 1781837616911093 us; error 60 us; skew 500 ppm
Jun 19 02:53:36 dist-test-slave-0f9d krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:36.915110 11532 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:53:36.915867 11532 webserver.cc:460] Webserver started at http://127.26.98.62:37833/ using document root <none> and password file <none>
I20260619 02:53:36.916162 11532 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:36.916242 11532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:36.916419 11532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:36.917869 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "8d0dfdc1168644249b0ab11bfa26373d"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-0f9d"
server_key: "56dcdc30cb27c93248cfa40c230ab67c"
server_key_iv: "826e1ac3e199b14330d4ae26136e4ceb"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.918334 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "8d0dfdc1168644249b0ab11bfa26373d"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-0f9d"
server_key: "56dcdc30cb27c93248cfa40c230ab67c"
server_key_iv: "826e1ac3e199b14330d4ae26136e4ceb"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.920354 11532 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:53:36.921738 11547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:36.922161 11532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:36.922415 11532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "8d0dfdc1168644249b0ab11bfa26373d"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-0f9d"
server_key: "56dcdc30cb27c93248cfa40c230ab67c"
server_key_iv: "826e1ac3e199b14330d4ae26136e4ceb"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.922605 11532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:36.938805 11532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:36.942076 11532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:36.942270 11532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:36.947249 11532 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:44963
I20260619 02:53:36.947302 11599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:44963 every 8 connection(s)
I20260619 02:53:36.947759 11532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:36.948975 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:36.952108 11600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: Bootstrap starting.
I20260619 02:53:36.952924 11600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:36.953248 11600 log.cc:824] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:36.954222 11600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: No bootstrap required, opened a new log
I20260619 02:53:36.956264 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11532
I20260619 02:53:36.956398 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:36.956660 27016 external_mini_cluster.cc:1428] Setting key 7cf6f61ae10de31862e58e2609209c56
I20260619 02:53:36.957137 11600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } }
I20260619 02:53:36.957459 11600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:36.957536 11600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0dfdc1168644249b0ab11bfa26373d, State: Initialized, Role: FOLLOWER
I20260619 02:53:36.957744 11600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } }
I20260619 02:53:36.957855 11600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:36.957914 11600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:36.957979 11600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:36.959488 11600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } }
I20260619 02:53:36.959656 11600 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0dfdc1168644249b0ab11bfa26373d; no voters:
I20260619 02:53:36.959892 11600 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:36.959942 11605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:36.960107 11605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [term 1 LEADER]: Becoming Leader. State: Replica: 8d0dfdc1168644249b0ab11bfa26373d, State: Running, Role: LEADER
I20260619 02:53:36.960299 11600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:36.960292 11605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } }
I20260619 02:53:36.961838 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0dfdc1168644249b0ab11bfa26373d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } } }
I20260619 02:53:36.962443 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:36.962028 11606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0dfdc1168644249b0ab11bfa26373d. Latest consensus state: current_term: 1 leader_uuid: "8d0dfdc1168644249b0ab11bfa26373d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dfdc1168644249b0ab11bfa26373d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44963 } } }
I20260619 02:53:36.962620 11606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:36.963085 11619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:36.963174 11619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:36.963250 11621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:36.963460 11621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:53:36 dist-test-slave-0f9d krb5kdc[11519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:36.965739 11621 catalog_manager.cc:1347] Generated new cluster ID: 83d69534a2d34cc0b739be9a3b77236c
I20260619 02:53:36.965817 11621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:36.970070 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:36.957614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.26.98.62:44963)
0619 02:53:36.957929 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:36.957967 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:36.958275 (+ 308us) server_negotiation.cc:407] Connection header received
0619 02:53:36.958461 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:36.958469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:36.958570 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:36.958743 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:36.959545 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.960839 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.962018 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.962763 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.962927 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:36.966106 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:36.966112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:36.966112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:36.966130 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:36.968155 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.968622 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.968628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.968629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.968769 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.968967 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.968969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.968969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.969331 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:36.969528 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:36.969613 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:53:36.969799 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":139,"threads_started":1}
I20260619 02:53:36.980361 11621 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:36.981323 11621 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:36.987092 11621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: Generated new TSK 0
I20260619 02:53:36.987386 11621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:53:37.022536 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:37.136813 11626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:37.137168 11626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:37.137204 11626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:37.139338 11626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:37.139454 11626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:37.139498 11626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:37.141788 11626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:37.142158 11626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:37.142439 11626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:37.149893 11631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.149893 11632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.149893 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.150784 11626 server_base.cc:1034] running on GCE node
I20260619 02:53:37.151129 11626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:37.151501 11626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:37.152885 11626 hybrid_clock.cc:648] HybridClock initialized: now 1781837617152828 us; error 44 us; skew 500 ppm
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:37.157181 11626 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:53:37.157761 11626 webserver.cc:460] Webserver started at http://127.26.98.1:43813/ using document root <none> and password file <none>
I20260619 02:53:37.157991 11626 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:37.158038 11626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:37.158160 11626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:37.159489 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "3fe48685c23847fbbc2b2c0b6308800d"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "17323e08d1d1884f404cca282e66e60f"
server_key_iv: "67522593c81de2b14b5a545b18d6092b"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.159917 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "3fe48685c23847fbbc2b2c0b6308800d"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "17323e08d1d1884f404cca282e66e60f"
server_key_iv: "67522593c81de2b14b5a545b18d6092b"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.161722 11626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:37.162670 11641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.162911 11626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:37.163012 11626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "3fe48685c23847fbbc2b2c0b6308800d"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "17323e08d1d1884f404cca282e66e60f"
server_key_iv: "67522593c81de2b14b5a545b18d6092b"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.163086 11626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:37.174638 11626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:37.177774 11626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:37.177933 11626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:37.178198 11626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:37.178548 11626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:37.178582 11626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.178608 11626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:37.178619 11626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.186626 11626 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:39363
I20260619 02:53:37.186676 11754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:39363 every 8 connection(s)
I20260619 02:53:37.187952 11626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:37.192232 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11626
I20260619 02:53:37.192421 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:37.192727 27016 external_mini_cluster.cc:1428] Setting key 3d181422fbfba2656a66e002044ccc25
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:53:37.203303 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.189295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:55011 (local address 127.26.98.62:44963)
0619 02:53:37.189396 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.189399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.189829 (+ 430us) server_negotiation.cc:407] Connection header received
0619 02:53:37.191385 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.191388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.191417 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.191502 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.192894 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.193780 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.195492 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.195749 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.195785 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.199181 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.199184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.199184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:37.199196 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.200885 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.201696 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.201698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.201698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.201774 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.202271 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.202272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.202272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.202431 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.202497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.203002 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.203115 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:53:37.203501 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.189199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44963 (local address 127.26.98.1:55011)
0619 02:53:37.189456 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.189593 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.191187 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.191638 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.191644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.192113 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:37.192645 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.192658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.193943 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.193945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.195319 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.195324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.195882 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.195882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.196001 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.196542 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:37.196551 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:37.198957 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:37.201042 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.201063 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.201068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.201543 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.201849 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.201850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.201850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.202135 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.202509 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:37.202522 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:37.202788 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.202959 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.203201 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260619 02:53:37.204195 11755 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44963
I20260619 02:53:37.204346 11755 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:37.204588 11755 heartbeater.cc:511] Master 127.26.98.62:44963 requested a full tablet report, sending...
I20260619 02:53:37.205439 11564 ts_manager.cc:194] Registered new tserver with Master: 3fe48685c23847fbbc2b2c0b6308800d (127.26.98.1:39363)
I20260619 02:53:37.206248 11564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:55011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:53:37.227181 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:37.330536 11760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:37.331046 11760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:37.331087 11760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:37.333685 11760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:37.333760 11760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:37.333796 11760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:37.335676 11760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:37.335965 11760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:37.336199 11760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:37.342541 11765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.342608 11768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.342557 11766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.342981 11760 server_base.cc:1034] running on GCE node
I20260619 02:53:37.343230 11760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:37.343544 11760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:37.344758 11760 hybrid_clock.cc:648] HybridClock initialized: now 1781837617344722 us; error 60 us; skew 500 ppm
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:37.353799 11760 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:53:37.354499 11760 webserver.cc:460] Webserver started at http://127.26.98.2:43805/ using document root <none> and password file <none>
I20260619 02:53:37.354784 11760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:37.354839 11760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:37.355024 11760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:37.356396 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "002d3a95405f4a039fb367469dfbd36a"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "65641387a862c24c7b1e252726901699"
server_key_iv: "59e4de1431ec72c4cbd425cc1c39647e"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.356882 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "002d3a95405f4a039fb367469dfbd36a"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "65641387a862c24c7b1e252726901699"
server_key_iv: "59e4de1431ec72c4cbd425cc1c39647e"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.358858 11760 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:37.360082 11775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.360396 11760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:37.360558 11760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "002d3a95405f4a039fb367469dfbd36a"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "65641387a862c24c7b1e252726901699"
server_key_iv: "59e4de1431ec72c4cbd425cc1c39647e"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.360669 11760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:37.369464 11760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:37.372752 11760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:37.372931 11760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:37.373219 11760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:37.373631 11760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:37.373680 11760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.373728 11760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:37.373747 11760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.381232 11760 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:36939
I20260619 02:53:37.381319 11888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:36939 every 8 connection(s)
I20260619 02:53:37.381790 11760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:37.384732 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11760
I20260619 02:53:37.384903 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:37.385161 27016 external_mini_cluster.cc:1428] Setting key 4f4e39ad8248e86651340f0d0cba3cb3
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:53:37.395582 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.382766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:60871 (local address 127.26.98.62:44963)
0619 02:53:37.382851 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.382853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.383142 (+ 289us) server_negotiation.cc:407] Connection header received
0619 02:53:37.383903 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.383905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.383924 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.383985 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.385604 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.386350 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.387854 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.388200 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.388299 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.391652 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.391654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.391654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:37.391665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.393541 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.394179 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.394181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.394181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.394251 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.394694 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.394695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.394696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.394852 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.394966 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.395325 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.395454 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:53:37.395780 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.382750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44963 (local address 127.26.98.2:60871)
0619 02:53:37.382979 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.383029 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.383730 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.384090 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.384096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.384883 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:37.385442 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.385455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.386552 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.386553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.387716 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.387722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.388336 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.388337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.388542 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.389310 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:37.389322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:37.391477 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:37.393701 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.393721 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.393726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.394071 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.394433 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.394435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.394435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.394606 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.394952 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:37.394959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:37.395163 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.395274 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.395512 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260619 02:53:37.396720 11889 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44963
I20260619 02:53:37.396848 11889 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:37.397089 11889 heartbeater.cc:511] Master 127.26.98.62:44963 requested a full tablet report, sending...
I20260619 02:53:37.397742 11564 ts_manager.cc:194] Registered new tserver with Master: 002d3a95405f4a039fb367469dfbd36a (127.26.98.2:36939)
I20260619 02:53:37.398169 11564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:60871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:53:37.418807 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:37.534109 11894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:37.534528 11894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:37.534577 11894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:37.536942 11894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:37.537057 11894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:37.537113 11894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:37.540254 11894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:37.540755 11894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:37.541119 11894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:37.549208 11900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.549571 11899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.549659 11894 server_base.cc:1034] running on GCE node
W20260619 02:53:37.551748 11902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.552304 11894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:37.552778 11894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:37.554049 11894 hybrid_clock.cc:648] HybridClock initialized: now 1781837617554025 us; error 46 us; skew 500 ppm
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:37.558225 11894 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:53:37.558966 11894 webserver.cc:460] Webserver started at http://127.26.98.3:36591/ using document root <none> and password file <none>
I20260619 02:53:37.559257 11894 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:37.559317 11894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:37.559466 11894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:37.560822 11894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "93448e0fbbb34a2f8f4fb2123729c3f1"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "12dc318d5a9d40d6b3286e6f6ee02cfc"
server_key_iv: "c751cf452eeff4965f4f4a63a47e4764"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.561268 11894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "93448e0fbbb34a2f8f4fb2123729c3f1"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "12dc318d5a9d40d6b3286e6f6ee02cfc"
server_key_iv: "c751cf452eeff4965f4f4a63a47e4764"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.562952 11894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:37.563892 11909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.564101 11894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:37.564225 11894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "93448e0fbbb34a2f8f4fb2123729c3f1"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-0f9d"
server_key: "12dc318d5a9d40d6b3286e6f6ee02cfc"
server_key_iv: "c751cf452eeff4965f4f4a63a47e4764"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.564340 11894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:37.579219 11894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:37.582413 11894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:37.582574 11894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:37.582870 11894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:37.583272 11894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:37.583310 11894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.583341 11894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:37.583359 11894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.591578 11894 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:45485
I20260619 02:53:37.591652 12022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:45485 every 8 connection(s)
I20260619 02:53:37.592069 11894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:37.596554 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 11894
I20260619 02:53:37.596695 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:37.596905 27016 external_mini_cluster.cc:1428] Setting key 38f61ba770b76afc9902444544ca06d6
Jun 19 02:53:37 dist-test-slave-0f9d krb5kdc[11519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:37.605590 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.593119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:56325 (local address 127.26.98.62:44963)
0619 02:53:37.593210 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.593214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.593445 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:53:37.594091 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.594094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.594111 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.594180 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.595613 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.596225 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.597685 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.597997 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.598070 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.600870 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.600874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.600874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:37.600888 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.602786 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.603830 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.603832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.603832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.603926 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.604446 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.604448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.604449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.604681 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.604790 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.605338 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.605498 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:37.605754 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.593021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:44963 (local address 127.26.98.3:56325)
0619 02:53:37.593300 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.593346 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.593969 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.594285 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.594292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.595148 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:37.595457 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.595465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.596353 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.596354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.597548 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.597554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.598093 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.598094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.598269 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.599106 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:37.599119 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:37.600711 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:37.603049 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.603066 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.603070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.603635 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.604067 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:37.604068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:37.604069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:37.604243 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:37.604858 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:37.604874 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:37.605088 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.605226 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.605470 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":134,"threads_started":1}
I20260619 02:53:37.606477 12023 heartbeater.cc:347] Connected to a master server at 127.26.98.62:44963
I20260619 02:53:37.606631 12023 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:37.606913 12023 heartbeater.cc:511] Master 127.26.98.62:44963 requested a full tablet report, sending...
I20260619 02:53:37.607496 11564 ts_manager.cc:194] Registered new tserver with Master: 93448e0fbbb34a2f8f4fb2123729c3f1 (127.26.98.3:45485)
I20260619 02:53:37.607962 11564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:56325
I20260619 02:53:37.608842 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:37.617172 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.610486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.26.98.62:44963)
0619 02:53:37.610554 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.610556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.610569 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:53:37.610652 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.610653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.610665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.610720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.611386 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.611937 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.612725 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.612929 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.612992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.614324 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.614326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.614327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:37.614337 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.615904 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.616301 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.616302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.616302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.616361 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:37.616531 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:37.616532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:37.616532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:37.616726 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.616833 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.616954 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.617056 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:37.619630 11564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:37.621569 11564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:37.630796 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.625844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39363 (local address 127.0.0.1:53016)
0619 02:53:37.626011 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.626064 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.626163 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.626717 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.626722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.626725 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:37.626842 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.626845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.628529 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.628530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.629573 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.629576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.630375 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.630493 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.630508 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.630587 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.630671 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260619 02:53:37.630925 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.626338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.26.98.2:36939)
0619 02:53:37.626548 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.626552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.626804 (+ 252us) server_negotiation.cc:407] Connection header received
0619 02:53:37.626894 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.626897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.626919 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.627011 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:37.627152 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.627998 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.629649 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.630466 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630554 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.630626 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.630733 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.630783 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260619 02:53:37.630800 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.626142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:36939 (local address 127.0.0.1:59342)
0619 02:53:37.626731 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.626742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.626802 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.627012 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.627013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.627016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:37.627075 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.627077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.628110 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.628111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.629495 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.629498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.630562 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.630639 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.630642 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.630702 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.630745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":81,"threads_started":1}
I20260619 02:53:37.631382 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.626332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:45485 (local address 127.0.0.1:41368)
0619 02:53:37.626895 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.626903 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.626959 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.627244 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.627245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.627248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:37.627303 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.627305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.628839 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.628841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.629909 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.629914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.631107 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.631108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.631177 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.631183 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.631238 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.631298 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":81,"threads_started":1}
I20260619 02:53:37.631486 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.626048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.26.98.1:39363)
0619 02:53:37.626342 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.626349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.626364 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:53:37.626433 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.626437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.626457 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.626565 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:37.627018 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.628330 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.629666 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.630271 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630348 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.630430 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.630863 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.631396 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":142,"threads_started":1}
I20260619 02:53:37.631636 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.626472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.26.98.3:45485)
0619 02:53:37.626677 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.626680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.627016 (+ 336us) server_negotiation.cc:407] Connection header received
0619 02:53:37.627068 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.627071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.627087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.627160 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:37.627391 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.628713 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630058 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.630919 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.630996 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.631062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.631489 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.631543 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
I20260619 02:53:37.632619 11957 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc6e5fab2564867ab236387d9eff08b (DEFAULT_TABLE table=test-table [id=1ed15ef41320475ca93a5ab973bf0aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:37.632614 11823 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc6e5fab2564867ab236387d9eff08b (DEFAULT_TABLE table=test-table [id=1ed15ef41320475ca93a5ab973bf0aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:37.632614 11689 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc6e5fab2564867ab236387d9eff08b (DEFAULT_TABLE table=test-table [id=1ed15ef41320475ca93a5ab973bf0aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:37.633047 11957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc6e5fab2564867ab236387d9eff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:37.633046 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc6e5fab2564867ab236387d9eff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:37.633046 11823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc6e5fab2564867ab236387d9eff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:37.636797 12040 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Bootstrap starting.
I20260619 02:53:37.637152 12041 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Bootstrap starting.
I20260619 02:53:37.637799 12041 tablet_bootstrap.cc:654] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:37.637801 12040 tablet_bootstrap.cc:654] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:37.638329 12041 log.cc:824] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:37.638330 12040 log.cc:824] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:37.638744 12042 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Bootstrap starting.
I20260619 02:53:37.639366 12040 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: No bootstrap required, opened a new log
I20260619 02:53:37.639431 12040 ts_tablet_manager.cc:1397] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:37.639525 12042 tablet_bootstrap.cc:654] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:37.640036 12042 log.cc:824] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:37.640938 12042 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: No bootstrap required, opened a new log
I20260619 02:53:37.641004 12042 ts_tablet_manager.cc:1397] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:37.641323 12040 raft_consensus.cc:348] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.641500 12040 raft_consensus.cc:374] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:37.641535 12040 raft_consensus.cc:729] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93448e0fbbb34a2f8f4fb2123729c3f1, State: Initialized, Role: FOLLOWER
I20260619 02:53:37.641671 12040 consensus_queue.cc:260] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.642025 12040 ts_tablet_manager.cc:1428] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:37.642060 12023 heartbeater.cc:503] Master 127.26.98.62:44963 was elected leader, sending a full tablet report...
I20260619 02:53:37.642660 12041 tablet_bootstrap.cc:492] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: No bootstrap required, opened a new log
I20260619 02:53:37.642761 12041 ts_tablet_manager.cc:1397] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 02:53:37.643102 12042 raft_consensus.cc:348] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.643324 12042 raft_consensus.cc:374] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:37.643362 12042 raft_consensus.cc:729] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe48685c23847fbbc2b2c0b6308800d, State: Initialized, Role: FOLLOWER
I20260619 02:53:37.643515 12042 consensus_queue.cc:260] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.643887 12042 ts_tablet_manager.cc:1428] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:37.643957 11755 heartbeater.cc:503] Master 127.26.98.62:44963 was elected leader, sending a full tablet report...
I20260619 02:53:37.644915 12041 raft_consensus.cc:348] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.645093 12041 raft_consensus.cc:374] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:37.645126 12041 raft_consensus.cc:729] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002d3a95405f4a039fb367469dfbd36a, State: Initialized, Role: FOLLOWER
I20260619 02:53:37.645282 12041 consensus_queue.cc:260] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.645653 12041 ts_tablet_manager.cc:1428] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:53:37.645673 11889 heartbeater.cc:503] Master 127.26.98.62:44963 was elected leader, sending a full tablet report...
W20260619 02:53:37.689380 11756 tablet.cc:2369] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:53:37.842933 12024 tablet.cc:2369] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:53:37.882820 11890 tablet.cc:2369] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:37.932227 12046 raft_consensus.cc:482] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:37.932425 12046 raft_consensus.cc:504] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.932919 12046 leader_election.cc:288] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fe48685c23847fbbc2b2c0b6308800d (127.26.98.1:39363), 002d3a95405f4a039fb367469dfbd36a (127.26.98.2:36939)
I20260619 02:53:37.937765 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.933366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:54715 (local address 127.26.98.1:39363)
0619 02:53:37.933430 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.933432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.933443 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:37.933568 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.933569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.933587 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.933656 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:37.933918 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.934744 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.936583 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.937218 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.937319 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.937335 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.937620 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.937681 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:37.937767 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.933587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:55193 (local address 127.26.98.2:36939)
0619 02:53:37.933653 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.933655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.933753 (+ 98us) server_negotiation.cc:407] Connection header received
0619 02:53:37.933808 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.933809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.933819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.933856 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:37.934029 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.934743 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.936564 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.937219 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.937319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.937335 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.937620 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.937681 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:37.937753 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.933298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39363 (local address 127.26.98.3:54715)
0619 02:53:37.933384 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.933399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.933459 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.933690 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.933692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.933697 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:37.933842 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.933845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.934911 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.934913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.936378 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.936383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.937358 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.937359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.937479 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.937482 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.937577 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.937628 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":66}
I20260619 02:53:37.937758 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.933461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:36939 (local address 127.26.98.3:55193)
0619 02:53:37.933670 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:37.933679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:37.933749 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:37.933879 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:37.933880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:37.933882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:37.933948 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.933949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.934911 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.934913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.936378 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:37.936383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.937358 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.937359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:37.937472 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.937479 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:53:37.937557 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:53:37.937618 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":105,"thread_start_us":161,"threads_started":1}
I20260619 02:53:37.938236 11709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc6e5fab2564867ab236387d9eff08b" candidate_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe48685c23847fbbc2b2c0b6308800d" is_pre_election: true
I20260619 02:53:37.938421 11709 raft_consensus.cc:2454] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93448e0fbbb34a2f8f4fb2123729c3f1 in term 0.
I20260619 02:53:37.938673 11913 leader_election.cc:302] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe48685c23847fbbc2b2c0b6308800d, 93448e0fbbb34a2f8f4fb2123729c3f1; no voters:
I20260619 02:53:37.938849 12046 raft_consensus.cc:2781] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:37.938819 11843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc6e5fab2564867ab236387d9eff08b" candidate_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002d3a95405f4a039fb367469dfbd36a" is_pre_election: true
I20260619 02:53:37.938926 12046 raft_consensus.cc:482] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:37.938959 12046 raft_consensus.cc:3037] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:37.939006 11843 raft_consensus.cc:2454] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93448e0fbbb34a2f8f4fb2123729c3f1 in term 0.
I20260619 02:53:37.939853 12046 raft_consensus.cc:504] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.940001 12046 leader_election.cc:288] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3fe48685c23847fbbc2b2c0b6308800d (127.26.98.1:39363), 002d3a95405f4a039fb367469dfbd36a (127.26.98.2:36939)
I20260619 02:53:37.940267 11709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc6e5fab2564867ab236387d9eff08b" candidate_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe48685c23847fbbc2b2c0b6308800d"
I20260619 02:53:37.940366 11843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc6e5fab2564867ab236387d9eff08b" candidate_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002d3a95405f4a039fb367469dfbd36a"
I20260619 02:53:37.940466 11709 raft_consensus.cc:3037] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:37.940562 11843 raft_consensus.cc:3037] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:37.941427 11709 raft_consensus.cc:2454] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93448e0fbbb34a2f8f4fb2123729c3f1 in term 1.
I20260619 02:53:37.941504 11843 raft_consensus.cc:2454] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93448e0fbbb34a2f8f4fb2123729c3f1 in term 1.
I20260619 02:53:37.941691 11913 leader_election.cc:302] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 002d3a95405f4a039fb367469dfbd36a, 93448e0fbbb34a2f8f4fb2123729c3f1; no voters:
I20260619 02:53:37.941835 12046 raft_consensus.cc:2781] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:37.942019 12046 raft_consensus.cc:686] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 93448e0fbbb34a2f8f4fb2123729c3f1, State: Running, Role: LEADER
I20260619 02:53:37.942133 12046 consensus_queue.cc:237] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } }
I20260619 02:53:37.943291 11564 catalog_manager.cc:5495] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93448e0fbbb34a2f8f4fb2123729c3f1 (127.26.98.3). New cstate: current_term: 1 leader_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93448e0fbbb34a2f8f4fb2123729c3f1" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45485 } health_report { overall_health: HEALTHY } } }
I20260619 02:53:37.990375 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.987560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.26.98.3:45485)
0619 02:53:37.987670 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.987672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.987684 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:37.987769 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.987770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.987802 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.987874 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:37.988037 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.988562 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.989372 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.989575 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.989637 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.989809 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:37.990053 (+ 244us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:37.990171 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.990215 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.990244 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:53:37.993453 11709 raft_consensus.cc:1261] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 1 FOLLOWER]: Refusing update from remote peer 93448e0fbbb34a2f8f4fb2123729c3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:37.993572 11843 raft_consensus.cc:1261] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 1 FOLLOWER]: Refusing update from remote peer 93448e0fbbb34a2f8f4fb2123729c3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:37.994029 12050 consensus_queue.cc:1035] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe48685c23847fbbc2b2c0b6308800d" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:37.994239 12046 consensus_queue.cc:1035] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002d3a95405f4a039fb367469dfbd36a" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:37.997566 12054 mvcc.cc:205] Tried to move back new op lower bound from 7298406883296321536 to 7298406883091726336. Current Snapshot: MvccSnapshot[applied={T|T < 7298406883296321536}]
I20260619 02:53:38.000600 11563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:38.000749 11563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:38.001778 11563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d0dfdc1168644249b0ab11bfa26373d: Sending DeleteTablet for 3 replicas of tablet 1bc6e5fab2564867ab236387d9eff08b
I20260619 02:53:38.002338 11823 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc6e5fab2564867ab236387d9eff08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:37 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:59342
I20260619 02:53:38.002466 11689 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc6e5fab2564867ab236387d9eff08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:37 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:53016
I20260619 02:53:38.002460 11957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc6e5fab2564867ab236387d9eff08b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:37 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:41368
I20260619 02:53:38.002724 12064 tablet_replica.cc:330] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d: stopping tablet replica
I20260619 02:53:38.002918 12064 raft_consensus.cc:2229] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:38.003103 12064 raft_consensus.cc:2258] T 1bc6e5fab2564867ab236387d9eff08b P 3fe48685c23847fbbc2b2c0b6308800d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:38.003293 12063 tablet_replica.cc:330] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: stopping tablet replica
I20260619 02:53:38.003409 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11626
I20260619 02:53:38.003433 12063 raft_consensus.cc:2229] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:38.004112 12063 raft_consensus.cc:2258] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:38.005265 12063 ts_tablet_manager.cc:1905] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:38.007098 12065 tablet_replica.cc:330] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: stopping tablet replica
I20260619 02:53:38.007387 12065 raft_consensus.cc:2229] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:38.007647 12065 raft_consensus.cc:2258] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:38.007827 12063 ts_tablet_manager.cc:1918] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:38.007988 12063 log.cc:1196] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/1bc6e5fab2564867ab236387d9eff08b
I20260619 02:53:38.008315 12063 ts_tablet_manager.cc:1939] T 1bc6e5fab2564867ab236387d9eff08b P 002d3a95405f4a039fb367469dfbd36a: Deleting consensus metadata
I20260619 02:53:38.008792 12065 ts_tablet_manager.cc:1905] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:38.009014 11551 catalog_manager.cc:4841] TS 002d3a95405f4a039fb367469dfbd36a (127.26.98.2:36939): tablet 1bc6e5fab2564867ab236387d9eff08b (table test-table [id=1ed15ef41320475ca93a5ab973bf0aef]) successfully deleted
W20260619 02:53:38.010368 11551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:39363 (error 108)
I20260619 02:53:38.010965 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.010799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39363 (local address 127.0.0.1:53018)
0619 02:53:38.010866 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.010928 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 02:53:38.011073 11551 catalog_manager.cc:4584] TS 3fe48685c23847fbbc2b2c0b6308800d (127.26.98.1:39363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bc6e5fab2564867ab236387d9eff08b: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39363: connect: Connection refused (error 111)
I20260619 02:53:38.011085 12065 ts_tablet_manager.cc:1918] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:38.011199 12065 log.cc:1196] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/1bc6e5fab2564867ab236387d9eff08b
I20260619 02:53:38.011384 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11760
I20260619 02:53:38.011511 12065 ts_tablet_manager.cc:1939] T 1bc6e5fab2564867ab236387d9eff08b P 93448e0fbbb34a2f8f4fb2123729c3f1: Deleting consensus metadata
I20260619 02:53:38.013198 11551 catalog_manager.cc:4841] TS 93448e0fbbb34a2f8f4fb2123729c3f1 (127.26.98.3:45485): tablet 1bc6e5fab2564867ab236387d9eff08b (table test-table [id=1ed15ef41320475ca93a5ab973bf0aef]) successfully deleted
I20260619 02:53:38.017663 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11894
I20260619 02:53:38.023341 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 11532
2026-06-19T02:53:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2379 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T02:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:38Z Disabled control of system clock
I20260619 02:53:38.056377 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40683
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:38.162992 12070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:38.163357 12070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:38.163386 12070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:38.165304 12070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:38.165413 12070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:38.165426 12070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:38.165439 12070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:38.165452 12070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:38.170194 12070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:40683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:38.170957 12070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:38.171530 12070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:38.178144 12076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.178151 12075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.178244 12070 server_base.cc:1034] running on GCE node
W20260619 02:53:38.178144 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.178815 12070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:38.179327 12070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:38.180547 12070 hybrid_clock.cc:648] HybridClock initialized: now 1781837618180519 us; error 48 us; skew 500 ppm
I20260619 02:53:38.183336 12070 webserver.cc:460] Webserver started at http://127.26.98.62:45939/ using document root <none> and password file <none>
I20260619 02:53:38.183760 12070 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:38.183837 12070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:38.184012 12070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:38.185734 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "0c212b9bcc75475e9a61d6f1c910e47d"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "d6204e9037d9a05323a7f306c379d248"
server_key_iv: "06bc4e72ffd28eacde403e96d5ad2469"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.186443 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "0c212b9bcc75475e9a61d6f1c910e47d"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "d6204e9037d9a05323a7f306c379d248"
server_key_iv: "06bc4e72ffd28eacde403e96d5ad2469"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.189724 12070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 02:53:38.191038 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.191470 12070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:53:38.191601 12070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0c212b9bcc75475e9a61d6f1c910e47d"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "d6204e9037d9a05323a7f306c379d248"
server_key_iv: "06bc4e72ffd28eacde403e96d5ad2469"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.191736 12070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:38.219092 12070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:38.219594 12070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:38.219759 12070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:38.225106 12070 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:40683
I20260619 02:53:38.225176 12136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:40683 every 8 connection(s)
I20260619 02:53:38.225665 12070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:38.227113 12137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:38.230479 12137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: Bootstrap starting.
I20260619 02:53:38.231371 12137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:38.231721 12137 log.cc:824] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:38.233238 12137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: No bootstrap required, opened a new log
I20260619 02:53:38.234709 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12070
I20260619 02:53:38.234858 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:38.235139 27016 external_mini_cluster.cc:1428] Setting key fc0a64ba1df38a79098dd92ce953f862
I20260619 02:53:38.235949 12137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } }
I20260619 02:53:38.236232 12137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:38.236274 12137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c212b9bcc75475e9a61d6f1c910e47d, State: Initialized, Role: FOLLOWER
I20260619 02:53:38.236441 12137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } }
I20260619 02:53:38.236529 12137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:38.236559 12137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:38.236624 12137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:38.237648 12137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } }
I20260619 02:53:38.237859 12137 leader_election.cc:302] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c212b9bcc75475e9a61d6f1c910e47d; no voters:
I20260619 02:53:38.238149 12137 leader_election.cc:288] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:38.238245 12142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:38.238391 12142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [term 1 LEADER]: Becoming Leader. State: Replica: 0c212b9bcc75475e9a61d6f1c910e47d, State: Running, Role: LEADER
I20260619 02:53:38.238510 12142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } }
I20260619 02:53:38.238566 12137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:38.239259 12143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } } }
I20260619 02:53:38.239553 12143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:38.240177 12151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:38.240343 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c212b9bcc75475e9a61d6f1c910e47d. Latest consensus state: current_term: 1 leader_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c212b9bcc75475e9a61d6f1c910e47d" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40683 } } }
I20260619 02:53:38.240422 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:38.240823 12151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:38.243356 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.236030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.26.98.62:40683)
0619 02:53:38.236419 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.236432 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.236476 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:53:38.236674 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.236685 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.236776 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.236966 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.237254 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.239048 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.240440 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.241219 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.241407 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.242385 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.242390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.242390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.242652 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.242710 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.242731 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.242816 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.242955 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.243071 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":157,"threads_started":1}
I20260619 02:53:38.243386 12151 catalog_manager.cc:1347] Generated new cluster ID: cb2da6d7f7e24a92baa0d19717dffbb8
I20260619 02:53:38.243469 12151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:38.253988 12151 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:38.254927 12151 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:38.263650 12151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: Generated new TSK 0
I20260619 02:53:38.263964 12151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:38.272764 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--builtin_ntp_servers=127.26.98.20:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:38.382455 12161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:38.382784 12161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:38.382823 12161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:38.384553 12161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:38.384629 12161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:38.384675 12161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:38.387337 12161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:38.387728 12161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:38.388114 12161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:38.394842 12169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.394865 12167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.394841 12161 server_base.cc:1034] running on GCE node
W20260619 02:53:38.394832 12166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.395332 12161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:38.395665 12161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:38.396868 12161 hybrid_clock.cc:648] HybridClock initialized: now 1781837618396796 us; error 86 us; skew 500 ppm
I20260619 02:53:38.398784 12161 webserver.cc:460] Webserver started at http://127.26.98.1:35455/ using document root <none> and password file <none>
I20260619 02:53:38.399153 12161 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:38.399230 12161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:38.399387 12161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:38.400861 12161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "380f30c8d9eb40a8b1c4231148a1a285"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "6402998e4305c2efdc0d035b7350b0b7"
server_key_iv: "d136f0d1e1a35b5b77525145490e23b7"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.401278 12161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "380f30c8d9eb40a8b1c4231148a1a285"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "6402998e4305c2efdc0d035b7350b0b7"
server_key_iv: "d136f0d1e1a35b5b77525145490e23b7"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.402920 12161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 02:53:38.403956 12175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.404198 12161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:38.404297 12161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "380f30c8d9eb40a8b1c4231148a1a285"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "6402998e4305c2efdc0d035b7350b0b7"
server_key_iv: "d136f0d1e1a35b5b77525145490e23b7"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.404382 12161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:38.425026 12161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:38.425439 12161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:38.425583 12161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:38.425843 12161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:38.426445 12161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:38.426493 12161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.426520 12161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:38.426532 12161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.434440 12161 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:33463
I20260619 02:53:38.434514 12288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:33463 every 8 connection(s)
I20260619 02:53:38.434966 12161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:38.440337 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12161
I20260619 02:53:38.440460 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:38.440656 27016 external_mini_cluster.cc:1428] Setting key 4e28b3a4692fe8c5f6272971597a9a9d
I20260619 02:53:38.441871 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.436032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40683 (local address 127.26.98.1:38609)
0619 02:53:38.436292 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.436334 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.436968 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.437414 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.437419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.437431 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:38.438132 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.438141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.438944 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.438945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.439863 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.439866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.440326 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.440327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.440462 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.441250 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:38.441251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:38.441293 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:38.441331 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:38.441507 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:38.441514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:38.441518 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.441588 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.441683 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260619 02:53:38.441902 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.436075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:38609 (local address 127.26.98.62:40683)
0619 02:53:38.436176 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.436179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.436428 (+ 249us) server_negotiation.cc:407] Connection header received
0619 02:53:38.437223 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.437226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.437250 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.437299 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.438229 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.438808 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.439963 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.440218 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.440290 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.441407 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.441408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.441409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.441415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.441431 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.441437 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.441483 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.441641 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.441681 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:53:38.442075 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--builtin_ntp_servers=127.26.98.20:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:38.442458 12289 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40683
I20260619 02:53:38.442606 12289 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:38.442951 12289 heartbeater.cc:511] Master 127.26.98.62:40683 requested a full tablet report, sending...
I20260619 02:53:38.444387 12100 ts_manager.cc:194] Registered new tserver with Master: 380f30c8d9eb40a8b1c4231148a1a285 (127.26.98.1:33463)
I20260619 02:53:38.445436 12100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.1:38609
W20260619 02:53:38.542186 12292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:38.542496 12292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:38.542572 12292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:38.544190 12292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:38.544258 12292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:38.544292 12292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:38.546033 12292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:38.546365 12292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:38.546733 12292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:38.553517 12298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.553525 12300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.553517 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.554060 12292 server_base.cc:1034] running on GCE node
I20260619 02:53:38.554350 12292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:38.554685 12292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:38.555879 12292 hybrid_clock.cc:648] HybridClock initialized: now 1781837618555798 us; error 102 us; skew 500 ppm
I20260619 02:53:38.557816 12292 webserver.cc:460] Webserver started at http://127.26.98.2:39929/ using document root <none> and password file <none>
I20260619 02:53:38.558125 12292 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:38.558192 12292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:38.558346 12292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:38.560118 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "95f511dc9a9f4e5f919acc3bd801eb64"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "4e2a7efd48f405d5b806b81ac2da0cf0"
server_key_iv: "171c38ff5fbb4f3a5f6f85b121f3ae92"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.560891 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "95f511dc9a9f4e5f919acc3bd801eb64"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "4e2a7efd48f405d5b806b81ac2da0cf0"
server_key_iv: "171c38ff5fbb4f3a5f6f85b121f3ae92"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.562719 12292 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 02:53:38.563870 12306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.564330 12292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:38.564450 12292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "95f511dc9a9f4e5f919acc3bd801eb64"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "4e2a7efd48f405d5b806b81ac2da0cf0"
server_key_iv: "171c38ff5fbb4f3a5f6f85b121f3ae92"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.564544 12292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:38.578359 12292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:38.578807 12292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:38.579006 12292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:38.579290 12292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:38.579663 12292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:38.579696 12292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.579728 12292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:38.579752 12292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.586615 12292 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:36253
I20260619 02:53:38.586663 12419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:36253 every 8 connection(s)
I20260619 02:53:38.587149 12292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:38.590080 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12292
I20260619 02:53:38.590188 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:38.590401 27016 external_mini_cluster.cc:1428] Setting key 640054d762de2fff922c9230e8f026da
I20260619 02:53:38.591588 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--builtin_ntp_servers=127.26.98.20:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:38.594003 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.588029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:60133 (local address 127.26.98.62:40683)
0619 02:53:38.588122 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.588124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.588348 (+ 224us) server_negotiation.cc:407] Connection header received
0619 02:53:38.589017 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.589019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.589033 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.589095 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.589911 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.590622 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.592200 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.592431 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.592499 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.593679 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.593681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.593682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.593689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.593710 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.593715 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.593765 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.593856 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.593902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:53:38.594305 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.588009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40683 (local address 127.26.98.2:60133)
0619 02:53:38.588216 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.588257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.588882 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.589176 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.589180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.589191 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:38.589809 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.589813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.590750 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.590751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.592078 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.592083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.592524 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.592525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.592726 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.593500 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:38.593501 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:38.593541 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:38.593575 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:38.593772 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:38.593779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:38.593783 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.593855 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.593992 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260619 02:53:38.595103 12420 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40683
I20260619 02:53:38.595261 12420 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:38.595510 12420 heartbeater.cc:511] Master 127.26.98.62:40683 requested a full tablet report, sending...
I20260619 02:53:38.596043 12100 ts_manager.cc:194] Registered new tserver with Master: 95f511dc9a9f4e5f919acc3bd801eb64 (127.26.98.2:36253)
I20260619 02:53:38.596534 12100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.2:60133
W20260619 02:53:38.691583 12423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:38.691911 12423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:38.691944 12423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:38.693607 12423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:38.693691 12423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:38.693725 12423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:38.695794 12423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:38.696166 12423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:38.696584 12423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:38.703049 12428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.703255 12431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.703423 12429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.704098 12423 server_base.cc:1034] running on GCE node
I20260619 02:53:38.704399 12423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:38.704751 12423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:38.705940 12423 hybrid_clock.cc:648] HybridClock initialized: now 1781837618705905 us; error 55 us; skew 500 ppm
I20260619 02:53:38.707757 12423 webserver.cc:460] Webserver started at http://127.26.98.3:42755/ using document root <none> and password file <none>
I20260619 02:53:38.708086 12423 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:38.708155 12423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:38.708317 12423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:38.709857 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "9a1dbdd2791f49debaf7ba311ac958de"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "39b2bed87f616297bbff8d622890b75b"
server_key_iv: "f435281bf4d15aa91e1a3e4cce6b846f"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.710366 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "9a1dbdd2791f49debaf7ba311ac958de"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "39b2bed87f616297bbff8d622890b75b"
server_key_iv: "f435281bf4d15aa91e1a3e4cce6b846f"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.712100 12423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 02:53:38.713176 12437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.713397 12423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:38.713510 12423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "9a1dbdd2791f49debaf7ba311ac958de"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-0f9d"
server_key: "39b2bed87f616297bbff8d622890b75b"
server_key_iv: "f435281bf4d15aa91e1a3e4cce6b846f"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.713610 12423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:38.734697 12423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:38.735225 12423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:38.735404 12423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:38.735708 12423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:38.736117 12423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:38.736157 12423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.736193 12423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:38.736209 12423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.744141 12423 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:46535
I20260619 02:53:38.744194 12550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:46535 every 8 connection(s)
I20260619 02:53:38.744910 12423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:38.749401 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12423
I20260619 02:53:38.749524 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:38.749778 27016 external_mini_cluster.cc:1428] Setting key 139894f2554b48bd91d5a74802ba9d71
I20260619 02:53:38.751818 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.746173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:35453 (local address 127.26.98.62:40683)
0619 02:53:38.746306 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.746309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.746590 (+ 281us) server_negotiation.cc:407] Connection header received
0619 02:53:38.747333 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.747335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.747352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.747409 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.748268 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.748843 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.750073 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.750298 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.750364 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.751465 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.751466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.751467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.751473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.751496 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.751502 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.751557 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.751652 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.751707 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 02:53:38.752090 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.746055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40683 (local address 127.26.98.3:35453)
0619 02:53:38.746397 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.746453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.747206 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.747498 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.747502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.747515 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:38.748161 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.748167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.748946 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.748948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.749978 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.749981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.750386 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.750387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.750563 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.751265 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:38.751266 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:38.751310 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:38.751344 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:38.751568 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:38.751575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:38.751580 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.751654 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.751809 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260619 02:53:38.752806 12551 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40683
I20260619 02:53:38.752938 12551 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:38.753188 12551 heartbeater.cc:511] Master 127.26.98.62:40683 requested a full tablet report, sending...
I20260619 02:53:38.753659 12100 ts_manager.cc:194] Registered new tserver with Master: 9a1dbdd2791f49debaf7ba311ac958de (127.26.98.3:46535)
I20260619 02:53:38.754112 12100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.3:35453
I20260619 02:53:38.761889 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:38.767328 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.763595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.26.98.62:40683)
0619 02:53:38.763672 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.763675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.763787 (+ 112us) server_negotiation.cc:407] Connection header received
0619 02:53:38.763873 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.763874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.763890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.763937 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.764202 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.764872 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.765775 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.765990 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.766098 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.767018 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.767021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.767021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.767029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.767051 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.767057 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.767101 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.767177 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.767223 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:53:38.769483 12100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:38.771634 12100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:38.780215 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.774793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:36253 (local address 127.0.0.1:44996)
0619 02:53:38.775127 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.775136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.775191 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.775731 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.775732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.775734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:38.775807 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.775809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.777116 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.777117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.778433 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.778436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779955 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.780017 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.780021 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.780080 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.780130 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":33,"thread_start_us":120,"threads_started":1}
I20260619 02:53:38.780180 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.775543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.26.98.1:33463)
0619 02:53:38.775811 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.775816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.775825 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:38.775868 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.775870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.775886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.775974 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:38.776074 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.776804 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.778580 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779378 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779458 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.779594 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.779996 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.780058 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260619 02:53:38.780335 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.774740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.26.98.3:46535)
0619 02:53:38.775064 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.775070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.775083 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:53:38.775138 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.775140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.775163 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.775260 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:38.775548 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.776789 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779188 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779753 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779824 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.779918 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.780182 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.780242 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":170,"threads_started":1}
I20260619 02:53:38.780432 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.775007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33463 (local address 127.0.0.1:45156)
0619 02:53:38.775265 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.775273 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.775311 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.775982 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.775983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.775984 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:38.776029 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.776030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.776921 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.776922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.778429 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.778433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779641 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.779760 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.779780 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.779850 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.780361 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":48,"thread_start_us":70,"threads_started":1}
I20260619 02:53:38.780499 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.774567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:46535 (local address 127.0.0.1:53642)
0619 02:53:38.774739 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.774779 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.774903 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.775355 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.775359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.775361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:38.775462 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.775464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.778191 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.778193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.779079 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.779083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779849 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.779932 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.779937 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.779987 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.780073 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260619 02:53:38.780814 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.774955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.26.98.2:36253)
0619 02:53:38.775381 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.775388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.775404 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:53:38.775462 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.775464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.775483 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.775632 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:38.775914 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.777013 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.778973 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.779765 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.779835 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.779921 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.780587 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.780714 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":247,"threads_started":1}
I20260619 02:53:38.781507 12223 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a (DEFAULT_TABLE table=test-table [id=6b0f4b7b480742bca00580743cfeb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:38.781713 12354 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a (DEFAULT_TABLE table=test-table [id=6b0f4b7b480742bca00580743cfeb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:38.782032 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1878fe9e1b47f296cea636b1cc8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:38.782056 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1878fe9e1b47f296cea636b1cc8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:38.783553 12485 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a (DEFAULT_TABLE table=test-table [id=6b0f4b7b480742bca00580743cfeb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:38.784041 12485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1878fe9e1b47f296cea636b1cc8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:38.785558 12568 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Bootstrap starting.
I20260619 02:53:38.786039 12569 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: Bootstrap starting.
I20260619 02:53:38.786626 12568 tablet_bootstrap.cc:654] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:38.786792 12569 tablet_bootstrap.cc:654] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:38.787187 12568 log.cc:824] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:38.787186 12569 log.cc:824] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:38.787813 12572 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Bootstrap starting.
I20260619 02:53:38.788122 12569 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: No bootstrap required, opened a new log
I20260619 02:53:38.788188 12569 ts_tablet_manager.cc:1397] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:38.788252 12568 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: No bootstrap required, opened a new log
I20260619 02:53:38.788321 12568 ts_tablet_manager.cc:1397] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:38.788684 12572 tablet_bootstrap.cc:654] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:38.789042 12572 log.cc:824] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:38.789930 12572 tablet_bootstrap.cc:492] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: No bootstrap required, opened a new log
I20260619 02:53:38.789994 12572 ts_tablet_manager.cc:1397] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:38.790550 12569 raft_consensus.cc:348] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.790711 12569 raft_consensus.cc:374] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:38.790735 12569 raft_consensus.cc:729] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380f30c8d9eb40a8b1c4231148a1a285, State: Initialized, Role: FOLLOWER
I20260619 02:53:38.790854 12569 consensus_queue.cc:260] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.791055 12568 raft_consensus.cc:348] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.791239 12568 raft_consensus.cc:374] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:38.791236 12569 ts_tablet_manager.cc:1428] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:38.791270 12568 raft_consensus.cc:729] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f511dc9a9f4e5f919acc3bd801eb64, State: Initialized, Role: FOLLOWER
I20260619 02:53:38.791397 12289 heartbeater.cc:503] Master 127.26.98.62:40683 was elected leader, sending a full tablet report...
I20260619 02:53:38.791394 12568 consensus_queue.cc:260] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.791688 12568 ts_tablet_manager.cc:1428] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:53:38.791877 12420 heartbeater.cc:503] Master 127.26.98.62:40683 was elected leader, sending a full tablet report...
I20260619 02:53:38.793852 12572 raft_consensus.cc:348] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.794059 12572 raft_consensus.cc:374] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:38.794083 12572 raft_consensus.cc:729] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1dbdd2791f49debaf7ba311ac958de, State: Initialized, Role: FOLLOWER
I20260619 02:53:38.794198 12572 consensus_queue.cc:260] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.794492 12572 ts_tablet_manager.cc:1428] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 02:53:38.794601 12551 heartbeater.cc:503] Master 127.26.98.62:40683 was elected leader, sending a full tablet report...
W20260619 02:53:38.837832 12421 tablet.cc:2369] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:38.848752 12575 raft_consensus.cc:482] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:38.848937 12575 raft_consensus.cc:504] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.849387 12575 leader_election.cc:288] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a1dbdd2791f49debaf7ba311ac958de (127.26.98.3:46535), 380f30c8d9eb40a8b1c4231148a1a285 (127.26.98.1:33463)
I20260619 02:53:38.854665 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.849680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:46535 (local address 127.26.98.2:49185)
0619 02:53:38.849776 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.849791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.849866 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.850058 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.850060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.850062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:38.850161 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.850164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.851238 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.851240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.853183 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.853189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.854225 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.854228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.854373 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.854382 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.854495 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.854546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":55}
I20260619 02:53:38.854671 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.849789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33463 (local address 127.26.98.2:42407)
0619 02:53:38.850005 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.850015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:38.850081 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:38.850205 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:38.850206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:38.850208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:38.850271 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.850273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.851239 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.851240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.853183 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:38.853189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.854225 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.854228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:38.854376 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.854382 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:53:38.854476 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:53:38.854545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":144,"thread_start_us":151,"threads_started":1}
I20260619 02:53:38.854734 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.849834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:49185 (local address 127.26.98.3:46535)
0619 02:53:38.849898 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.849900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.849912 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:38.849953 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.849954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.849965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.850028 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:38.850237 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.851092 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.853361 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.854034 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.854134 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.854151 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.854559 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.854650 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:38.854734 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.849914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:42407 (local address 127.26.98.1:33463)
0619 02:53:38.849963 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.849965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.850094 (+ 129us) server_negotiation.cc:407] Connection header received
0619 02:53:38.850127 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.850128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.850141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.850185 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:38.850334 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.851092 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.853361 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.854022 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.854147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.854159 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.854559 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.854650 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:53:38.855129 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1878fe9e1b47f296cea636b1cc8e1a" candidate_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380f30c8d9eb40a8b1c4231148a1a285" is_pre_election: true
I20260619 02:53:38.855176 12505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1878fe9e1b47f296cea636b1cc8e1a" candidate_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1dbdd2791f49debaf7ba311ac958de" is_pre_election: true
I20260619 02:53:38.855342 12505 raft_consensus.cc:2454] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f511dc9a9f4e5f919acc3bd801eb64 in term 0.
I20260619 02:53:38.855397 12243 raft_consensus.cc:2454] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f511dc9a9f4e5f919acc3bd801eb64 in term 0.
I20260619 02:53:38.855645 12308 leader_election.cc:302] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f511dc9a9f4e5f919acc3bd801eb64, 9a1dbdd2791f49debaf7ba311ac958de; no voters:
I20260619 02:53:38.855886 12575 raft_consensus.cc:2781] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:38.855996 12575 raft_consensus.cc:482] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:38.856027 12575 raft_consensus.cc:3037] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:38.857286 12575 raft_consensus.cc:504] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.857465 12575 leader_election.cc:288] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [CANDIDATE]: Term 1 election: Requested vote from peers 9a1dbdd2791f49debaf7ba311ac958de (127.26.98.3:46535), 380f30c8d9eb40a8b1c4231148a1a285 (127.26.98.1:33463)
I20260619 02:53:38.857926 12505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1878fe9e1b47f296cea636b1cc8e1a" candidate_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a1dbdd2791f49debaf7ba311ac958de"
I20260619 02:53:38.858055 12505 raft_consensus.cc:3037] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:38.859078 12505 raft_consensus.cc:2454] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f511dc9a9f4e5f919acc3bd801eb64 in term 1.
I20260619 02:53:38.859282 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1878fe9e1b47f296cea636b1cc8e1a" candidate_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380f30c8d9eb40a8b1c4231148a1a285"
I20260619 02:53:38.859347 12308 leader_election.cc:302] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f511dc9a9f4e5f919acc3bd801eb64, 9a1dbdd2791f49debaf7ba311ac958de; no voters:
I20260619 02:53:38.859412 12243 raft_consensus.cc:3037] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:38.859524 12575 raft_consensus.cc:2781] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:38.859737 12575 raft_consensus.cc:686] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 1 LEADER]: Becoming Leader. State: Replica: 95f511dc9a9f4e5f919acc3bd801eb64, State: Running, Role: LEADER
I20260619 02:53:38.859863 12575 consensus_queue.cc:237] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } }
I20260619 02:53:38.860479 12243 raft_consensus.cc:2454] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f511dc9a9f4e5f919acc3bd801eb64 in term 1.
I20260619 02:53:38.861043 12101 catalog_manager.cc:5495] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95f511dc9a9f4e5f919acc3bd801eb64 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95f511dc9a9f4e5f919acc3bd801eb64" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 } health_report { overall_health: UNKNOWN } } }
I20260619 02:53:38.876920 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.873746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.26.98.2:36253)
0619 02:53:38.873809 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.873811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.873817 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:53:38.873863 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.873864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.873889 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.873948 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:38.874184 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.874730 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.875686 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:38.875916 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:38.876005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:38.876259 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:38.876416 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:38.876505 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.876730 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.876758 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:38.879582 12505 raft_consensus.cc:1261] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 1 FOLLOWER]: Refusing update from remote peer 95f511dc9a9f4e5f919acc3bd801eb64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:38.879582 12243 raft_consensus.cc:1261] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285 [term 1 FOLLOWER]: Refusing update from remote peer 95f511dc9a9f4e5f919acc3bd801eb64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:38.880196 12575 consensus_queue.cc:1035] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a1dbdd2791f49debaf7ba311ac958de" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 46535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:38.880402 12578 consensus_queue.cc:1035] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "380f30c8d9eb40a8b1c4231148a1a285" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:38.884146 12581 mvcc.cc:205] Tried to move back new op lower bound from 7298406886925897728 to 7298406886850564096. Current Snapshot: MvccSnapshot[applied={T|T < 7298406886925897728}]
I20260619 02:53:38.884330 12582 mvcc.cc:205] Tried to move back new op lower bound from 7298406886925897728 to 7298406886850564096. Current Snapshot: MvccSnapshot[applied={T|T < 7298406886925897728}]
I20260619 02:53:38.884500 12583 mvcc.cc:205] Tried to move back new op lower bound from 7298406886925897728 to 7298406886850564096. Current Snapshot: MvccSnapshot[applied={T|T < 7298406886925897728}]
I20260619 02:53:38.888156 12100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:38.888314 12100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:38.889361 12100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c212b9bcc75475e9a61d6f1c910e47d: Sending DeleteTablet for 3 replicas of tablet 9e1878fe9e1b47f296cea636b1cc8e1a
I20260619 02:53:38.889918 12354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:38 UTC) from {username='slave'} at 127.0.0.1:44996
I20260619 02:53:38.890085 12485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:38 UTC) from {username='slave'} at 127.0.0.1:53642
I20260619 02:53:38.890060 12223 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1878fe9e1b47f296cea636b1cc8e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:38 UTC) from {username='slave'} at 127.0.0.1:45156
I20260619 02:53:38.890389 12592 tablet_replica.cc:330] T 9e1878fe9e1b47f296cea636b1cc8e1a P 380f30c8d9eb40a8b1c4231148a1a285: stopping tablet replica
I20260619 02:53:38.890522 12591 tablet_replica.cc:330] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: stopping tablet replica
I20260619 02:53:38.890631 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12161
I20260619 02:53:38.891523 12591 raft_consensus.cc:2229] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:38.891805 12591 raft_consensus.cc:2258] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:38.892652 12591 ts_tablet_manager.cc:1905] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:38.892722 12593 tablet_replica.cc:330] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: stopping tablet replica
I20260619 02:53:38.892879 12593 raft_consensus.cc:2229] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:38.893013 12593 raft_consensus.cc:2258] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:38.893596 12593 ts_tablet_manager.cc:1905] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:38.895264 12591 ts_tablet_manager.cc:1918] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:38.895452 12591 log.cc:1196] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/9e1878fe9e1b47f296cea636b1cc8e1a
I20260619 02:53:38.895567 12593 ts_tablet_manager.cc:1918] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:38.895670 12593 log.cc:1196] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/9e1878fe9e1b47f296cea636b1cc8e1a
I20260619 02:53:38.895968 12591 ts_tablet_manager.cc:1939] T 9e1878fe9e1b47f296cea636b1cc8e1a P 95f511dc9a9f4e5f919acc3bd801eb64: Deleting consensus metadata
I20260619 02:53:38.895993 12593 ts_tablet_manager.cc:1939] T 9e1878fe9e1b47f296cea636b1cc8e1a P 9a1dbdd2791f49debaf7ba311ac958de: Deleting consensus metadata
I20260619 02:53:38.896678 12086 catalog_manager.cc:4841] TS 95f511dc9a9f4e5f919acc3bd801eb64 (127.26.98.2:36253): tablet 9e1878fe9e1b47f296cea636b1cc8e1a (table test-table [id=6b0f4b7b480742bca00580743cfeb0af]) successfully deleted
I20260619 02:53:38.896819 12086 catalog_manager.cc:4841] TS 9a1dbdd2791f49debaf7ba311ac958de (127.26.98.3:46535): tablet 9e1878fe9e1b47f296cea636b1cc8e1a (table test-table [id=6b0f4b7b480742bca00580743cfeb0af]) successfully deleted
W20260619 02:53:38.897985 12086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:33463 (error 108)
I20260619 02:53:38.898747 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.898558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:33463 (local address 127.0.0.1:45164)
0619 02:53:38.898618 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:38.898702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 02:53:38.898912 12086 catalog_manager.cc:4584] TS 380f30c8d9eb40a8b1c4231148a1a285 (127.26.98.1:33463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e1878fe9e1b47f296cea636b1cc8e1a: Network error: Client connection negotiation failed: client connection to 127.26.98.1:33463: connect: Connection refused (error 111)
I20260619 02:53:38.899250 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12292
I20260619 02:53:38.904979 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12423
I20260619 02:53:38.910491 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12070
2026-06-19T02:53:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (886 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:38 dist-test-slave-0f9d krb5kdc[12597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:38 dist-test-slave-0f9d krb5kdc[12597](info): set up 2 sockets
Jun 19 02:53:38 dist-test-slave-0f9d krb5kdc[12597](info): commencing operation
krb5kdc: starting...
W20260619 02:53:39.919410 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:53:40.058017 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36041
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:40.169672 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:40.170037 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:40.170069 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:40.172112 12610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:40.172200 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:40.172216 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:40.172250 12610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:40.172264 12610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:40.174343 12610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:40.174711 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:40.175040 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:40.181697 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.181960 12610 server_base.cc:1034] running on GCE node
W20260619 02:53:40.181697 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.181697 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.182446 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:40.182929 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:40.184181 12610 hybrid_clock.cc:648] HybridClock initialized: now 1781837620184149 us; error 56 us; skew 500 ppm
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:40.188663 12610 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:53:40.189507 12610 webserver.cc:460] Webserver started at http://127.26.98.62:38863/ using document root <none> and password file <none>
I20260619 02:53:40.189817 12610 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:40.189888 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:40.190066 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:40.191565 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "f51d17e50e3e48adb74ebe118d8d0ef9"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "89bb521ab05a76803fbb84493e78e0ee"
server_key_iv: "b2476a94b9a3c19c52c06a6d40567b85"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.192070 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "f51d17e50e3e48adb74ebe118d8d0ef9"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "89bb521ab05a76803fbb84493e78e0ee"
server_key_iv: "b2476a94b9a3c19c52c06a6d40567b85"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.193917 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:40.195399 12625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.195751 12610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:40.195883 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "f51d17e50e3e48adb74ebe118d8d0ef9"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "89bb521ab05a76803fbb84493e78e0ee"
server_key_iv: "b2476a94b9a3c19c52c06a6d40567b85"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.195994 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:40.207320 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:40.210664 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:40.210866 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:40.216459 12610 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36041
I20260619 02:53:40.216639 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36041 every 8 connection(s)
I20260619 02:53:40.216979 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:40.218043 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:40.221266 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: Bootstrap starting.
I20260619 02:53:40.222123 12678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:40.222470 12678 log.cc:824] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:40.223529 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: No bootstrap required, opened a new log
I20260619 02:53:40.225797 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12610
I20260619 02:53:40.225644 12678 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } }
I20260619 02:53:40.225937 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:40.225945 12678 raft_consensus.cc:374] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:40.225993 12678 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51d17e50e3e48adb74ebe118d8d0ef9, State: Initialized, Role: FOLLOWER
I20260619 02:53:40.226187 27016 external_mini_cluster.cc:1428] Setting key a39178309a705caa1591ae631452cac4
I20260619 02:53:40.226207 12678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } }
I20260619 02:53:40.226312 12678 raft_consensus.cc:388] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:40.226357 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:40.226416 12678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:40.227977 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } }
I20260619 02:53:40.228166 12678 leader_election.cc:302] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51d17e50e3e48adb74ebe118d8d0ef9; no voters:
I20260619 02:53:40.228412 12678 leader_election.cc:288] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:40.228463 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:40.228627 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [term 1 LEADER]: Becoming Leader. State: Replica: f51d17e50e3e48adb74ebe118d8d0ef9, State: Running, Role: LEADER
I20260619 02:53:40.228766 12678 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:40.228756 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } }
I20260619 02:53:40.229341 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } } }
I20260619 02:53:40.229449 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:40.229607 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51d17e50e3e48adb74ebe118d8d0ef9. Latest consensus state: current_term: 1 leader_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51d17e50e3e48adb74ebe118d8d0ef9" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36041 } } }
I20260619 02:53:40.229671 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:40.230866 12692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:40.232060 12692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:40.234216 12692 catalog_manager.cc:1347] Generated new cluster ID: b9ebd9e41db74c338deefc5794b1d142
I20260619 02:53:40.234323 12692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:40.244232 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.226924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.26.98.62:36041)
0619 02:53:40.227165 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.227176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.227210 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:40.227355 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.227362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.227421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.227585 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.228505 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.230695 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.232963 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.234370 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.234550 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.238522 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.238531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.238531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.238558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.241760 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.242456 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.242462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.242463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.242659 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.242920 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.242921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.242922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.243399 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.243622 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.243732 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.243925 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260619 02:53:40.246358 12692 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:40.247397 12692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:40.254395 12692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: Generated new TSK 0
I20260619 02:53:40.254678 12692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:53:40.288389 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:40.394012 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:40.394320 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:40.394366 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:40.397119 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:40.397257 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:40.397316 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:40.399973 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:40.400390 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:40.400702 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:40.407035 12704 server_base.cc:1034] running on GCE node
W20260619 02:53:40.406975 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.406975 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.406975 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.407526 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:40.407823 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:40.409080 12704 hybrid_clock.cc:648] HybridClock initialized: now 1781837620409056 us; error 38 us; skew 500 ppm
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:40.412699 12704 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:53:40.413391 12704 webserver.cc:460] Webserver started at http://127.26.98.1:33149/ using document root <none> and password file <none>
I20260619 02:53:40.413707 12704 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:40.413772 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:40.413934 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:40.415333 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "ccb00a88430c475e95f1eeca262c8ece"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "c10aeef9a8848df7a965e7d42888606b"
server_key_iv: "72db9d110ae3353b48b97110138bb823"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.415761 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "ccb00a88430c475e95f1eeca262c8ece"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "c10aeef9a8848df7a965e7d42888606b"
server_key_iv: "72db9d110ae3353b48b97110138bb823"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.417516 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:40.418565 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.418852 12704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:40.419075 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "ccb00a88430c475e95f1eeca262c8ece"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "c10aeef9a8848df7a965e7d42888606b"
server_key_iv: "72db9d110ae3353b48b97110138bb823"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.419173 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:40.433180 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:40.436864 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:40.437027 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:40.437289 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:40.437638 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:40.437674 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.437711 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:40.437724 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.445027 12704 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:34991
I20260619 02:53:40.445092 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:34991 every 8 connection(s)
I20260619 02:53:40.445540 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:40.446060 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12704
I20260619 02:53:40.446179 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:40.446439 27016 external_mini_cluster.cc:1428] Setting key eb20c4d382aea7dd834fcdfe02a24a41
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
I20260619 02:53:40.459304 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.446771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:54443 (local address 127.26.98.62:36041)
0619 02:53:40.446834 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.446836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.447162 (+ 326us) server_negotiation.cc:407] Connection header received
0619 02:53:40.447843 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.447844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.447865 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.447925 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.449400 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.449972 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.451751 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.452002 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.452102 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.455510 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.455513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.455514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.455525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.457317 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.457999 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.458002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.458002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.458089 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.458399 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.458400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.458400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.458546 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.458603 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.459077 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.459215 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:53:40.459631 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.446777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36041 (local address 127.26.98.1:54443)
0619 02:53:40.447003 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.447057 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.447714 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.448023 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.448028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.448804 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:40.449218 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.449229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.450124 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.450125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.451561 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.451568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.452160 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.452161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.452394 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.453185 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:40.453201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:40.455333 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:40.457465 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.457481 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.457486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.457880 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.458159 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.458160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.458160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.458316 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.458602 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:40.458608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:40.458814 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.459032 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.459333 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
I20260619 02:53:40.460247 12833 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36041
I20260619 02:53:40.460371 12833 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:40.460610 12833 heartbeater.cc:511] Master 127.26.98.62:36041 requested a full tablet report, sending...
I20260619 02:53:40.461318 12640 ts_manager.cc:194] Registered new tserver with Master: ccb00a88430c475e95f1eeca262c8ece (127.26.98.1:34991)
I20260619 02:53:40.462148 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:54443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:53:40.482431 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:40.581388 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:40.581897 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:40.581988 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:40.584686 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:40.584802 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:40.584846 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:40.587518 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:40.587922 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:40.588495 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:40.594710 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.594691 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.594823 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.595263 12838 server_base.cc:1034] running on GCE node
I20260619 02:53:40.595575 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:40.595896 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:40.597087 12838 hybrid_clock.cc:648] HybridClock initialized: now 1781837620597046 us; error 63 us; skew 500 ppm
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:40.600663 12838 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:53:40.601265 12838 webserver.cc:460] Webserver started at http://127.26.98.2:46867/ using document root <none> and password file <none>
I20260619 02:53:40.601568 12838 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:40.601640 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:40.601823 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:40.603243 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "c7103b82a9bc457ea148506381acabb2"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "84c06288325d28c3e7343625273ead2f"
server_key_iv: "eb07b2d455db8dd254f095c80bf66af2"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.603680 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "c7103b82a9bc457ea148506381acabb2"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "84c06288325d28c3e7343625273ead2f"
server_key_iv: "eb07b2d455db8dd254f095c80bf66af2"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.605355 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:40.606452 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.606657 12838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:40.606760 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "c7103b82a9bc457ea148506381acabb2"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "84c06288325d28c3e7343625273ead2f"
server_key_iv: "eb07b2d455db8dd254f095c80bf66af2"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.606845 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:40.625792 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:40.629402 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:40.629614 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:40.630111 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:40.630775 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:40.630831 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.630868 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:40.630913 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.638473 12838 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:45035
I20260619 02:53:40.638550 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:45035 every 8 connection(s)
I20260619 02:53:40.639006 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:40.639927 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12838
I20260619 02:53:40.640174 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:40.640583 27016 external_mini_cluster.cc:1428] Setting key aeea48a2187702e9cd1e1c0f0d148705
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
I20260619 02:53:40.654009 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.640251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:52133 (local address 127.26.98.62:36041)
0619 02:53:40.640336 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.640339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.640576 (+ 237us) server_negotiation.cc:407] Connection header received
0619 02:53:40.641344 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.641346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.641360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.641458 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.642817 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.643598 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.645636 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.645945 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.646055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.649029 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.649032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.649033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.649047 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.651251 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.652189 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.652192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.652192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.652310 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.652790 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.652791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.652792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.653001 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.653102 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.653771 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.653918 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:53:40.654315 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.640110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36041 (local address 127.26.98.2:52133)
0619 02:53:40.640400 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.640458 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.641187 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.641622 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.641629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.642350 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:40.642670 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.642676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.643769 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.643771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.645436 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.645443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.646114 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.646116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.646297 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.647076 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:40.647092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:40.648819 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:40.651431 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.651447 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.651451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.652010 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.652456 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.652458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.652459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.652667 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.653173 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:40.653185 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:40.653487 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.653635 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.653986 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":113,"threads_started":1}
I20260619 02:53:40.655369 12967 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36041
I20260619 02:53:40.655529 12967 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:40.655786 12967 heartbeater.cc:511] Master 127.26.98.62:36041 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
I20260619 02:53:40.656558 12640 ts_manager.cc:194] Registered new tserver with Master: c7103b82a9bc457ea148506381acabb2 (127.26.98.2:45035)
I20260619 02:53:40.657152 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:52133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:53:40.675027 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:40.775233 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:40.775522 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:40.775563 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:40.777108 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:40.777179 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:40.777227 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:40.779197 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:40.779517 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:40.779788 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:40.785176 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.785203 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.785188 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.785581 12972 server_base.cc:1034] running on GCE node
I20260619 02:53:40.785820 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:40.786098 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:40.787263 12972 hybrid_clock.cc:648] HybridClock initialized: now 1781837620787244 us; error 35 us; skew 500 ppm
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:40.795146 12972 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:53:40.795773 12972 webserver.cc:460] Webserver started at http://127.26.98.3:42837/ using document root <none> and password file <none>
I20260619 02:53:40.796046 12972 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:40.796104 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:40.796244 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:40.797559 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "44e3647584434cdda22701c5ea73c6e5"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "bff6f7ec68e4e7743034afb9ff791e04"
server_key_iv: "fbb398164e9f067e55a2e60bb0fa493b"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.797955 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "44e3647584434cdda22701c5ea73c6e5"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "bff6f7ec68e4e7743034afb9ff791e04"
server_key_iv: "fbb398164e9f067e55a2e60bb0fa493b"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.799671 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:53:40.800870 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.801461 12972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:40.801589 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "44e3647584434cdda22701c5ea73c6e5"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-0f9d"
server_key: "bff6f7ec68e4e7743034afb9ff791e04"
server_key_iv: "fbb398164e9f067e55a2e60bb0fa493b"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.801714 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:40.819052 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:40.822500 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:40.822752 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:40.823100 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:40.823496 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:40.823534 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.823568 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:40.823592 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.831028 12972 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:33131
I20260619 02:53:40.831084 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:33131 every 8 connection(s)
I20260619 02:53:40.831504 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:40.832417 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 12972
I20260619 02:53:40.832538 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:40.832862 27016 external_mini_cluster.cc:1428] Setting key 95dcddc642cecd5e1a1e8593d553342e
Jun 19 02:53:40 dist-test-slave-0f9d krb5kdc[12597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:40.846004 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.832524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59429 (local address 127.26.98.62:36041)
0619 02:53:40.832597 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.832601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.832891 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:53:40.833548 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.833548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.833564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.833624 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.834958 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.835499 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.836934 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.837157 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.837241 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.840983 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.840986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.840986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.840997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.842865 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.844083 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.844085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.844086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.844186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.844870 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.844880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.844881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.845088 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.845176 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.845730 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.845923 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:53:40.846145 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.832510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36041 (local address 127.26.98.3:59429)
0619 02:53:40.832699 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.832751 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.833439 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.833704 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.833709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.834474 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:40.834762 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.834770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.835704 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.835713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.836772 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.836777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.837383 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.837385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.837579 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.838348 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:40.838362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:40.840759 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:40.843059 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.843076 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.843080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.843900 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.844357 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:40.844358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:40.844358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:40.844744 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:40.845225 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:40.845255 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:40.845506 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.845647 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.845908 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260619 02:53:40.847029 13101 heartbeater.cc:347] Connected to a master server at 127.26.98.62:36041
I20260619 02:53:40.847213 13101 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:40.847455 13101 heartbeater.cc:511] Master 127.26.98.62:36041 requested a full tablet report, sending...
I20260619 02:53:40.848112 12640 ts_manager.cc:194] Registered new tserver with Master: 44e3647584434cdda22701c5ea73c6e5 (127.26.98.3:33131)
I20260619 02:53:40.848565 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:59429
I20260619 02:53:40.857640 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:40.868016 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.859497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.26.98.62:36041)
0619 02:53:40.859653 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.859657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.859874 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:53:40.859957 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.859958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.859973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.860029 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.860713 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.861296 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.862132 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.862342 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.862409 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.864315 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.864318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.864318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.864328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.866137 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.866716 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.866718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.866718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.866785 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.867047 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.867048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.867048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.867233 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.867345 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.867628 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.867739 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 02:53:40.870612 12640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:40.873051 12640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:40.881737 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.26.98.2:45035)
0619 02:53:40.877946 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.877953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.877965 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:40.878004 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.878006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.878019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.878102 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:40.878271 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.879477 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.880615 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.881220 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.881297 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.881380 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.881571 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.881628 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260619 02:53:40.881844 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:33131 (local address 127.0.0.1:35886)
0619 02:53:40.877329 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.877478 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.877558 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.878026 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.878030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.878034 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:40.878138 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.878141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.879213 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.879214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.880474 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.880477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.881533 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.881534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.881618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.881623 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.881667 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.881719 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260619 02:53:40.882054 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.26.98.3:33131)
0619 02:53:40.877851 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.877855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.877866 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:40.877915 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.877917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.877933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.878017 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:40.878243 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.879091 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.880580 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.881407 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.881774 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.881859 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.881887 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.881950 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":108,"threads_started":1}
I20260619 02:53:40.881724 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:45035 (local address 127.0.0.1:47614)
0619 02:53:40.877635 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.877645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.877724 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.878128 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.878129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.878131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:40.878196 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.878198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.879639 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.879640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.880520 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.880522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.881333 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.881335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.881442 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.881459 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.881540 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.881615 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
I20260619 02:53:40.883306 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:34991 (local address 127.0.0.1:52776)
0619 02:53:40.877411 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.877725 (+ 314us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.877781 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.878129 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.878130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.878132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:40.878183 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.878185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.880593 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.880594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.881684 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.881686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.883026 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.883027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.883100 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.883105 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.883159 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.883213 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260619 02:53:40.883332 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.877273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.26.98.1:34991)
0619 02:53:40.877488 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.877494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.877841 (+ 347us) server_negotiation.cc:407] Connection header received
0619 02:53:40.877904 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.877908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.877928 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.878014 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:40.878247 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.879477 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.881789 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.882935 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.883026 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.883094 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.883187 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.883238 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260619 02:53:40.883236 13035 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa8f82854094c20aa3015125c3233db (DEFAULT_TABLE table=test-table [id=ffa9b9aadc1646ca99ead124b01059e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:40.883256 12901 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa8f82854094c20aa3015125c3233db (DEFAULT_TABLE table=test-table [id=ffa9b9aadc1646ca99ead124b01059e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:40.883733 13035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa8f82854094c20aa3015125c3233db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:40.883733 12901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa8f82854094c20aa3015125c3233db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:40.884549 12767 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa8f82854094c20aa3015125c3233db (DEFAULT_TABLE table=test-table [id=ffa9b9aadc1646ca99ead124b01059e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:40.885020 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa8f82854094c20aa3015125c3233db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:40.888398 13118 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Bootstrap starting.
I20260619 02:53:40.888937 13120 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Bootstrap starting.
I20260619 02:53:40.889134 13119 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Bootstrap starting.
I20260619 02:53:40.889564 13118 tablet_bootstrap.cc:654] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:40.889595 13120 tablet_bootstrap.cc:654] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:40.890017 13119 tablet_bootstrap.cc:654] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:40.890229 13120 log.cc:824] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:40.890229 13118 log.cc:824] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:40.890650 13119 log.cc:824] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:40.891199 13118 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: No bootstrap required, opened a new log
I20260619 02:53:40.891268 13120 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: No bootstrap required, opened a new log
I20260619 02:53:40.891287 13118 ts_tablet_manager.cc:1397] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:40.891350 13120 ts_tablet_manager.cc:1397] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:40.891737 13119 tablet_bootstrap.cc:492] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: No bootstrap required, opened a new log
I20260619 02:53:40.891822 13119 ts_tablet_manager.cc:1397] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:53:40.892903 13118 raft_consensus.cc:348] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.893324 13118 raft_consensus.cc:374] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:40.893370 13118 raft_consensus.cc:729] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb00a88430c475e95f1eeca262c8ece, State: Initialized, Role: FOLLOWER
I20260619 02:53:40.893587 13118 consensus_queue.cc:260] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.894022 13118 ts_tablet_manager.cc:1428] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:40.894198 12833 heartbeater.cc:503] Master 127.26.98.62:36041 was elected leader, sending a full tablet report...
I20260619 02:53:40.894368 13119 raft_consensus.cc:348] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.894594 13119 raft_consensus.cc:374] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:40.894637 13119 raft_consensus.cc:729] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e3647584434cdda22701c5ea73c6e5, State: Initialized, Role: FOLLOWER
I20260619 02:53:40.894761 13119 consensus_queue.cc:260] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.895176 13119 ts_tablet_manager.cc:1428] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:53:40.894547 13120 raft_consensus.cc:348] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.895325 13120 raft_consensus.cc:374] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:40.895370 13120 raft_consensus.cc:729] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7103b82a9bc457ea148506381acabb2, State: Initialized, Role: FOLLOWER
I20260619 02:53:40.895491 13120 consensus_queue.cc:260] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.895569 13101 heartbeater.cc:503] Master 127.26.98.62:36041 was elected leader, sending a full tablet report...
I20260619 02:53:40.895797 13120 ts_tablet_manager.cc:1428] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:53:40.895931 12967 heartbeater.cc:503] Master 127.26.98.62:36041 was elected leader, sending a full tablet report...
I20260619 02:53:40.931176 13124 raft_consensus.cc:482] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:40.931356 13124 raft_consensus.cc:504] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.931782 13124 leader_election.cc:288] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e3647584434cdda22701c5ea73c6e5 (127.26.98.3:33131), c7103b82a9bc457ea148506381acabb2 (127.26.98.2:45035)
I20260619 02:53:40.934952 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.932146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:36545 (local address 127.26.98.3:33131)
0619 02:53:40.932198 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.932200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.932209 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:40.932259 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.932260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.932273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.932323 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:40.932481 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.933033 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934142 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.934553 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934624 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.934638 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.934792 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.934837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:53:40.934953 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.932100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:57493 (local address 127.26.98.2:45035)
0619 02:53:40.932164 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.932167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.932299 (+ 132us) server_negotiation.cc:407] Connection header received
0619 02:53:40.932356 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.932357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.932369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.932420 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:40.932575 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.933084 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934132 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.934559 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934624 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.934639 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.934782 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.934826 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:40.934962 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.932030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:33131 (local address 127.26.98.1:36545)
0619 02:53:40.932099 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.932114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.932178 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.932332 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.932333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.932334 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:40.932418 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.932420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.933124 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.933125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.934032 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.934036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.934635 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.934726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.934730 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.934776 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.934827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":41}
I20260619 02:53:40.934970 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.932061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:45035 (local address 127.26.98.1:57493)
0619 02:53:40.932213 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.932235 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:40.932299 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:40.932427 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:40.932427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:40.932429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:40.932510 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.932511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.933163 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.933164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.934041 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:40.934043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.934640 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.934641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:40.934719 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.934724 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:40.934770 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:53:40.934822 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260619 02:53:40.935395 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa8f82854094c20aa3015125c3233db" candidate_uuid: "ccb00a88430c475e95f1eeca262c8ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7103b82a9bc457ea148506381acabb2" is_pre_election: true
I20260619 02:53:40.935411 13055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa8f82854094c20aa3015125c3233db" candidate_uuid: "ccb00a88430c475e95f1eeca262c8ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e3647584434cdda22701c5ea73c6e5" is_pre_election: true
I20260619 02:53:40.935582 12921 raft_consensus.cc:2454] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb00a88430c475e95f1eeca262c8ece in term 0.
I20260619 02:53:40.935582 13055 raft_consensus.cc:2454] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb00a88430c475e95f1eeca262c8ece in term 0.
I20260619 02:53:40.935847 12721 leader_election.cc:302] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7103b82a9bc457ea148506381acabb2, ccb00a88430c475e95f1eeca262c8ece; no voters:
I20260619 02:53:40.936007 13124 raft_consensus.cc:2781] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:40.936084 13124 raft_consensus.cc:482] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:40.936106 13124 raft_consensus.cc:3037] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:40.936986 13124 raft_consensus.cc:504] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.937147 13124 leader_election.cc:288] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [CANDIDATE]: Term 1 election: Requested vote from peers 44e3647584434cdda22701c5ea73c6e5 (127.26.98.3:33131), c7103b82a9bc457ea148506381acabb2 (127.26.98.2:45035)
I20260619 02:53:40.937357 13055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa8f82854094c20aa3015125c3233db" candidate_uuid: "ccb00a88430c475e95f1eeca262c8ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e3647584434cdda22701c5ea73c6e5"
I20260619 02:53:40.937384 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa8f82854094c20aa3015125c3233db" candidate_uuid: "ccb00a88430c475e95f1eeca262c8ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7103b82a9bc457ea148506381acabb2"
I20260619 02:53:40.937464 12921 raft_consensus.cc:3037] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:40.937453 13055 raft_consensus.cc:3037] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:40.938411 12921 raft_consensus.cc:2454] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb00a88430c475e95f1eeca262c8ece in term 1.
I20260619 02:53:40.938475 13055 raft_consensus.cc:2454] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb00a88430c475e95f1eeca262c8ece in term 1.
I20260619 02:53:40.938577 12721 leader_election.cc:302] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7103b82a9bc457ea148506381acabb2, ccb00a88430c475e95f1eeca262c8ece; no voters:
I20260619 02:53:40.938778 13124 raft_consensus.cc:2781] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:40.939013 13124 raft_consensus.cc:686] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 1 LEADER]: Becoming Leader. State: Replica: ccb00a88430c475e95f1eeca262c8ece, State: Running, Role: LEADER
I20260619 02:53:40.939139 13124 consensus_queue.cc:237] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } }
I20260619 02:53:40.940141 12640 catalog_manager.cc:5495] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb00a88430c475e95f1eeca262c8ece (127.26.98.1). New cstate: current_term: 1 leader_uuid: "ccb00a88430c475e95f1eeca262c8ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb00a88430c475e95f1eeca262c8ece" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 } health_report { overall_health: UNKNOWN } } }
W20260619 02:53:40.946811 12834 tablet.cc:2369] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:40.959725 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.956848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.26.98.1:34991)
0619 02:53:40.956917 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.956919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.956930 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:40.956980 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.956981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.957007 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.957086 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:40.957244 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.957792 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.958729 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.958993 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.959083 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.959289 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:40.959427 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:40.959539 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.959601 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.959614 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:40.962502 12921 raft_consensus.cc:1261] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 1 FOLLOWER]: Refusing update from remote peer ccb00a88430c475e95f1eeca262c8ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:40.962528 13055 raft_consensus.cc:1261] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 1 FOLLOWER]: Refusing update from remote peer ccb00a88430c475e95f1eeca262c8ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:40.963148 13124 consensus_queue.cc:1035] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e3647584434cdda22701c5ea73c6e5" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:40.963406 13128 consensus_queue.cc:1035] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7103b82a9bc457ea148506381acabb2" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:40.969771 12640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:40.969939 12640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:40.971189 12640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f51d17e50e3e48adb74ebe118d8d0ef9: Sending DeleteTablet for 3 replicas of tablet 1fa8f82854094c20aa3015125c3233db
I20260619 02:53:40.971731 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa8f82854094c20aa3015125c3233db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:40 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:52776
I20260619 02:53:40.971774 12901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa8f82854094c20aa3015125c3233db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:40 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:47614
I20260619 02:53:40.971864 13035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa8f82854094c20aa3015125c3233db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:40 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:35886
I20260619 02:53:40.972060 13141 tablet_replica.cc:330] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece: stopping tablet replica
I20260619 02:53:40.972205 13141 raft_consensus.cc:2229] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:40.972474 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12704
I20260619 02:53:40.972615 13141 raft_consensus.cc:2258] T 1fa8f82854094c20aa3015125c3233db P ccb00a88430c475e95f1eeca262c8ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:40.973277 13142 tablet_replica.cc:330] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: stopping tablet replica
I20260619 02:53:40.973449 13142 raft_consensus.cc:2229] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:40.973587 13142 raft_consensus.cc:2258] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:40.973807 13143 tablet_replica.cc:330] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: stopping tablet replica
I20260619 02:53:40.973944 13143 raft_consensus.cc:2229] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:40.974076 13143 raft_consensus.cc:2258] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:40.974860 13143 ts_tablet_manager.cc:1905] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:40.975888 13142 ts_tablet_manager.cc:1905] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:40.978068 13142 ts_tablet_manager.cc:1918] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:40.978082 13143 ts_tablet_manager.cc:1918] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:40.978258 13143 log.cc:1196] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/1fa8f82854094c20aa3015125c3233db
I20260619 02:53:40.978282 13142 log.cc:1196] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/1fa8f82854094c20aa3015125c3233db
I20260619 02:53:40.978595 13143 ts_tablet_manager.cc:1939] T 1fa8f82854094c20aa3015125c3233db P 44e3647584434cdda22701c5ea73c6e5: Deleting consensus metadata
I20260619 02:53:40.978611 13142 ts_tablet_manager.cc:1939] T 1fa8f82854094c20aa3015125c3233db P c7103b82a9bc457ea148506381acabb2: Deleting consensus metadata
I20260619 02:53:40.979182 12626 catalog_manager.cc:4841] TS 44e3647584434cdda22701c5ea73c6e5 (127.26.98.3:33131): tablet 1fa8f82854094c20aa3015125c3233db (table test-table [id=ffa9b9aadc1646ca99ead124b01059e9]) successfully deleted
I20260619 02:53:40.979204 12627 catalog_manager.cc:4841] TS c7103b82a9bc457ea148506381acabb2 (127.26.98.2:45035): tablet 1fa8f82854094c20aa3015125c3233db (table test-table [id=ffa9b9aadc1646ca99ead124b01059e9]) successfully deleted
W20260619 02:53:40.981974 12627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:34991 (error 108)
I20260619 02:53:40.982301 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12838
I20260619 02:53:40.983078 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.982451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:34991 (local address 127.0.0.1:52800)
0619 02:53:40.982980 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:40.983032 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
W20260619 02:53:40.983408 12627 catalog_manager.cc:4584] TS ccb00a88430c475e95f1eeca262c8ece (127.26.98.1:34991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fa8f82854094c20aa3015125c3233db: Network error: Client connection negotiation failed: client connection to 127.26.98.1:34991: connect: Connection refused (error 111)
I20260619 02:53:40.987591 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12972
I20260619 02:53:40.992612 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 12610
2026-06-19T02:53:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T02:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:41Z Disabled control of system clock
I20260619 02:53:41.024495 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39639
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:41.127069 13148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:41.127378 13148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:41.127403 13148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:41.129743 13148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:41.129864 13148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:41.129877 13148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:41.129890 13148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:41.129901 13148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:41.132299 13148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:41.132675 13148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:41.133024 13148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:41.140023 13153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.140167 13156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.140023 13154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.141005 13148 server_base.cc:1034] running on GCE node
I20260619 02:53:41.141412 13148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:41.141817 13148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:41.143184 13148 hybrid_clock.cc:648] HybridClock initialized: now 1781837621143151 us; error 56 us; skew 500 ppm
I20260619 02:53:41.145149 13148 webserver.cc:460] Webserver started at http://127.26.98.62:44799/ using document root <none> and password file <none>
I20260619 02:53:41.145462 13148 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:41.145526 13148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:41.145694 13148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:41.147305 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "688489de571a4d7eb04e136df30bb499"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "8b81c950d69f04509b485aac9e86c364"
server_key_iv: "bb34681bb87dd355c94af54527abbd39"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.147783 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "688489de571a4d7eb04e136df30bb499"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "8b81c950d69f04509b485aac9e86c364"
server_key_iv: "bb34681bb87dd355c94af54527abbd39"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.149850 13148 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:41.153075 13162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.153879 13148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:41.154299 13148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "688489de571a4d7eb04e136df30bb499"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "8b81c950d69f04509b485aac9e86c364"
server_key_iv: "bb34681bb87dd355c94af54527abbd39"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.154712 13148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:41.178797 13148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:41.179342 13148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:41.179508 13148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:41.185457 13148 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:39639
I20260619 02:53:41.185524 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:39639 every 8 connection(s)
I20260619 02:53:41.186010 13148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:41.187829 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:41.191111 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: Bootstrap starting.
I20260619 02:53:41.191938 13215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:41.192315 13215 log.cc:824] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:41.192806 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13148
I20260619 02:53:41.192931 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:41.193189 27016 external_mini_cluster.cc:1428] Setting key a1abe37afcb52e7ab1627086b4ace94e
I20260619 02:53:41.193388 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: No bootstrap required, opened a new log
I20260619 02:53:41.195883 13215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } }
I20260619 02:53:41.196117 13215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:41.196153 13215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688489de571a4d7eb04e136df30bb499, State: Initialized, Role: FOLLOWER
I20260619 02:53:41.196348 13215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } }
I20260619 02:53:41.196462 13215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:41.196494 13215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:41.196548 13215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:41.197374 13215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } }
I20260619 02:53:41.197516 13215 leader_election.cc:302] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688489de571a4d7eb04e136df30bb499; no voters:
I20260619 02:53:41.197744 13215 leader_election.cc:288] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:41.197850 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:41.198060 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [term 1 LEADER]: Becoming Leader. State: Replica: 688489de571a4d7eb04e136df30bb499, State: Running, Role: LEADER
I20260619 02:53:41.198144 13215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:41.198225 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } }
I20260619 02:53:41.200493 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688489de571a4d7eb04e136df30bb499. Latest consensus state: current_term: 1 leader_uuid: "688489de571a4d7eb04e136df30bb499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } } }
I20260619 02:53:41.201339 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:41.200731 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688489de571a4d7eb04e136df30bb499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688489de571a4d7eb04e136df30bb499" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39639 } } }
I20260619 02:53:41.201463 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499 [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:41.200991 13235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:41.201536 13235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:41.201680 13237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:41.201910 13237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:41.203045 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.193874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.26.98.62:39639)
0619 02:53:41.194130 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.194212 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.194263 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:53:41.195833 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.195846 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.195951 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.196159 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.196471 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.198368 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.200142 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.200893 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.201074 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.202025 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.202031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.202031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.202292 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.202350 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.202371 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.202453 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.202599 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.202730 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260619 02:53:41.204312 13237 catalog_manager.cc:1347] Generated new cluster ID: c6adce4102324c1badb5646bc5936103
I20260619 02:53:41.204381 13237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:41.217295 13237 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:41.219259 13237 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:41.228152 13237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: Generated new TSK 0
I20260619 02:53:41.228513 13237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:41.239203 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--builtin_ntp_servers=127.26.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:41.349368 13239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:41.349725 13239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:41.349756 13239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:41.351595 13239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:41.351696 13239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:41.351734 13239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:41.353791 13239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:41.354182 13239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:41.354640 13239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:41.361990 13245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.362100 13244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.362354 13239 server_base.cc:1034] running on GCE node
W20260619 02:53:41.362003 13247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.362959 13239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:41.363473 13239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:41.364709 13239 hybrid_clock.cc:648] HybridClock initialized: now 1781837621364691 us; error 55 us; skew 500 ppm
I20260619 02:53:41.367441 13239 webserver.cc:460] Webserver started at http://127.26.98.1:34873/ using document root <none> and password file <none>
I20260619 02:53:41.367825 13239 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:41.367901 13239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:41.368064 13239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:41.369920 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "e94b8a28489f45fc85cbf42b477243ec"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "745ff893b1043dfafb7cb4bc37019be2"
server_key_iv: "5607b6ca2c5e9fe664316706c0bbfa15"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.370492 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "e94b8a28489f45fc85cbf42b477243ec"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "745ff893b1043dfafb7cb4bc37019be2"
server_key_iv: "5607b6ca2c5e9fe664316706c0bbfa15"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.372658 13239 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260619 02:53:41.373867 13253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.374191 13239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:41.374320 13239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "e94b8a28489f45fc85cbf42b477243ec"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "745ff893b1043dfafb7cb4bc37019be2"
server_key_iv: "5607b6ca2c5e9fe664316706c0bbfa15"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.374433 13239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:41.389631 13239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:41.390103 13239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:41.390260 13239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:41.390568 13239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:41.391045 13239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:41.391086 13239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.391113 13239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:41.391136 13239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.399041 13239 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:39625
I20260619 02:53:41.399127 13366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:39625 every 8 connection(s)
I20260619 02:53:41.399652 13239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:41.407169 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13239
I20260619 02:53:41.407229 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.400718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:42525 (local address 127.26.98.62:39639)
0619 02:53:41.400802 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.400805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.401196 (+ 391us) server_negotiation.cc:407] Connection header received
0619 02:53:41.401906 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.401908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.401926 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.401997 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.403027 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.403614 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.404999 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.405261 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.405344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.406630 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.406633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.406633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.406641 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.406669 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.406677 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.406779 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.407027 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.407088 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:41.407301 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:41.407481 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.400716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39639 (local address 127.26.98.1:42525)
0619 02:53:41.401027 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.401083 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.401764 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.402086 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.402091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.402103 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:41.402822 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.402830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.403794 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.403796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.404825 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.404831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.405398 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.405400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.405577 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.406357 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:41.406358 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:41.406402 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:41.406438 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:41.406822 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:41.406833 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:41.406838 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.406980 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.407160 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":153,"threads_started":1}
I20260619 02:53:41.407563 27016 external_mini_cluster.cc:1428] Setting key 5e75d2b99b2e17d0d1569e961d2bb1c8
I20260619 02:53:41.408320 13367 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39639
I20260619 02:53:41.408531 13367 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:41.408946 13367 heartbeater.cc:511] Master 127.26.98.62:39639 requested a full tablet report, sending...
I20260619 02:53:41.409046 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--builtin_ntp_servers=127.26.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:41.409924 13178 ts_manager.cc:194] Registered new tserver with Master: e94b8a28489f45fc85cbf42b477243ec (127.26.98.1:39625)
I20260619 02:53:41.410979 13178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.1:42525
W20260619 02:53:41.511987 13370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:41.512315 13370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:41.512344 13370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:41.513883 13370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:41.513955 13370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:41.513988 13370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:41.515781 13370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:41.516121 13370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:41.516494 13370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:41.522825 13378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.522964 13375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.522825 13376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.523273 13370 server_base.cc:1034] running on GCE node
I20260619 02:53:41.523584 13370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:41.523910 13370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:41.525234 13370 hybrid_clock.cc:648] HybridClock initialized: now 1781837621525197 us; error 36 us; skew 500 ppm
I20260619 02:53:41.527094 13370 webserver.cc:460] Webserver started at http://127.26.98.2:41743/ using document root <none> and password file <none>
I20260619 02:53:41.527433 13370 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:41.527501 13370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:41.527637 13370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:41.528991 13370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "b03714d62f224215b670189d11d4b4b9"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "64f7d7b546803cfc397751650dd1ce49"
server_key_iv: "60371091d6072255e2c186417418f090"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.529409 13370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "b03714d62f224215b670189d11d4b4b9"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "64f7d7b546803cfc397751650dd1ce49"
server_key_iv: "60371091d6072255e2c186417418f090"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.531109 13370 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:53:41.532356 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.532748 13370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:41.532886 13370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "b03714d62f224215b670189d11d4b4b9"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "64f7d7b546803cfc397751650dd1ce49"
server_key_iv: "60371091d6072255e2c186417418f090"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.532989 13370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:41.558449 13370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:41.559022 13370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:41.559230 13370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:41.559537 13370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:41.560006 13370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:41.560055 13370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.560089 13370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:41.560107 13370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.567304 13370 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40689
I20260619 02:53:41.567451 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40689 every 8 connection(s)
I20260619 02:53:41.567821 13370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:41.574848 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.568843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:49433 (local address 127.26.98.62:39639)
0619 02:53:41.568959 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.568963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.569185 (+ 222us) server_negotiation.cc:407] Connection header received
0619 02:53:41.569910 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.569913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.569929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.570058 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.571257 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.571806 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.573088 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.573317 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.573399 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.574468 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.574470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.574471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.574478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.574497 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.574502 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.574560 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.574694 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.574760 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:53:41.575133 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.568742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39639 (local address 127.26.98.2:49433)
0619 02:53:41.569014 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.569069 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.569733 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.570234 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.570340 (+ 106us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.570357 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:41.571071 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.571081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.571954 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.571955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.572949 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.572952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.573452 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.573453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.573616 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.574273 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:41.574274 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:41.574314 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:41.574347 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:41.574601 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:41.574609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:41.574613 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.574692 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.574809 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260619 02:53:41.575767 13498 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39639
I20260619 02:53:41.575884 13498 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:41.576123 13498 heartbeater.cc:511] Master 127.26.98.62:39639 requested a full tablet report, sending...
I20260619 02:53:41.576555 13178 ts_manager.cc:194] Registered new tserver with Master: b03714d62f224215b670189d11d4b4b9 (127.26.98.2:40689)
I20260619 02:53:41.576968 13178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.2:49433
I20260619 02:53:41.577165 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13370
I20260619 02:53:41.577271 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:41.577505 27016 external_mini_cluster.cc:1428] Setting key 4eddfd9f6caa16d6135d7b4f27fbe463
I20260619 02:53:41.578547 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--builtin_ntp_servers=127.26.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:41.680253 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:41.680670 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:41.680718 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:41.682658 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:41.682749 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:41.682786 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:41.684885 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:41.685271 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:41.685664 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:41.691855 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.691856 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.691855 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.692190 13501 server_base.cc:1034] running on GCE node
I20260619 02:53:41.692615 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:41.692955 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:41.694203 13501 hybrid_clock.cc:648] HybridClock initialized: now 1781837621694176 us; error 42 us; skew 500 ppm
I20260619 02:53:41.696250 13501 webserver.cc:460] Webserver started at http://127.26.98.3:45519/ using document root <none> and password file <none>
I20260619 02:53:41.696768 13501 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:41.696856 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:41.696987 13501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:41.698520 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "a55afa1233e747a288acb71bbe67829f"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "89efb0018aabf1bf50c5063093bb5cc1"
server_key_iv: "256047951b23798e5382c7ae4d775aac"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.699011 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "a55afa1233e747a288acb71bbe67829f"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "89efb0018aabf1bf50c5063093bb5cc1"
server_key_iv: "256047951b23798e5382c7ae4d775aac"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.700903 13501 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:41.702047 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.702454 13501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:41.702594 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "a55afa1233e747a288acb71bbe67829f"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-0f9d"
server_key: "89efb0018aabf1bf50c5063093bb5cc1"
server_key_iv: "256047951b23798e5382c7ae4d775aac"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.702692 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:41.738095 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:41.738517 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:41.738667 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:41.738983 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:41.739372 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:41.739409 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.739435 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:41.739446 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.747210 13501 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:45947
I20260619 02:53:41.747355 13628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:45947 every 8 connection(s)
I20260619 02:53:41.747797 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:41.754344 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.748862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:43263 (local address 127.26.98.62:39639)
0619 02:53:41.748977 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.748979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.749208 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:53:41.749794 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.749796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.749811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.749878 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.750705 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.751320 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.752499 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.752728 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.752794 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.753978 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.753980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.753981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.753988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.754012 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.754017 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.754081 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.754209 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.754254 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 02:53:41.754666 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.748792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39639 (local address 127.26.98.3:43263)
0619 02:53:41.749037 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.749087 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.749675 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.749966 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.749970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.749980 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:41.750585 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.750589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.751437 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.751439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.752389 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.752393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.752836 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.752838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.752991 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.753762 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:41.753763 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:41.753806 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:41.753845 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:41.754115 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:41.754122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:41.754126 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.754207 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.754339 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260619 02:53:41.755414 13629 heartbeater.cc:347] Connected to a master server at 127.26.98.62:39639
I20260619 02:53:41.755577 13629 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:41.755820 13629 heartbeater.cc:511] Master 127.26.98.62:39639 requested a full tablet report, sending...
I20260619 02:53:41.756363 13178 ts_manager.cc:194] Registered new tserver with Master: a55afa1233e747a288acb71bbe67829f (127.26.98.3:45947)
I20260619 02:53:41.756773 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13501
I20260619 02:53:41.756831 13178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.3:43263
I20260619 02:53:41.756904 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:41.757108 27016 external_mini_cluster.cc:1428] Setting key a3c59a2ba081db957aef2c1ab99176eb
I20260619 02:53:41.758155 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:41.762840 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.759694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.26.98.62:39639)
0619 02:53:41.759745 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.759747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.759890 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:53:41.759959 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.759959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.759978 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.760033 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.760189 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.760722 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.761492 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.761700 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.761759 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.762542 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.762544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.762545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.762551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.762567 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.762571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.762647 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.762723 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.762736 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:41.764811 13178 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:41.766983 13178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:41.775368 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.770849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:45947 (local address 127.0.0.1:40828)
0619 02:53:41.771104 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.771112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.771153 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.771843 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.771844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.771846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:41.771891 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.771893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.773404 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.773405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.774152 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.774155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.774977 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.774978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.775073 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.775091 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.775179 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.775259 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260619 02:53:41.775381 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.771161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.26.98.3:45947)
0619 02:53:41.771621 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.771627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.771638 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:41.771685 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.771688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.771704 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.771840 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:41.772297 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.773321 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.774251 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.774840 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.774972 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.775122 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.775198 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.775259 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":112,"threads_started":1}
I20260619 02:53:41.775672 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.770546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40689 (local address 127.0.0.1:46800)
0619 02:53:41.770762 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.770811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.770938 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.771459 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.771460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.771463 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:41.771521 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.771524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.772780 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.772782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.774440 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.774442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.775404 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.775405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.775476 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.775481 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.775528 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.775585 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":31,"thread_start_us":119,"threads_started":1}
I20260619 02:53:41.775951 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.770671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.26.98.2:40689)
0619 02:53:41.771279 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.771285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.771298 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:53:41.771344 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.771346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.771364 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.771460 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:41.771640 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.772681 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.774527 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.775203 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.775276 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.775366 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.775799 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.775857 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":95,"threads_started":1}
I20260619 02:53:41.776237 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.770584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39625 (local address 127.0.0.1:47944)
0619 02:53:41.770908 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.770929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.770997 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.771340 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.771346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.771350 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:41.771499 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.771502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.772972 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.772973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.774154 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.774157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.775997 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.775998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.776073 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.776079 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.776110 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.776161 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":8,"thread_start_us":116,"threads_started":1}
I20260619 02:53:41.776750 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.770801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.26.98.1:39625)
0619 02:53:41.771105 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.771111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.771125 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:53:41.771176 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.771178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.771196 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.771304 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:41.771610 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.772882 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.774974 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.775803 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.776424 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.776570 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.776601 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.776656 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":138,"threads_started":1}
I20260619 02:53:41.776983 13432 tablet_service.cc:1467] Processing CreateTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b (DEFAULT_TABLE table=test-table [id=0ecd8d692ea949289d9e6a68283a71ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:41.776985 13563 tablet_service.cc:1467] Processing CreateTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b (DEFAULT_TABLE table=test-table [id=0ecd8d692ea949289d9e6a68283a71ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:41.777484 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fdfef98416437db0fbc9dfa8f71a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:41.777585 13563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fdfef98416437db0fbc9dfa8f71a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:41.777535 13301 tablet_service.cc:1467] Processing CreateTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b (DEFAULT_TABLE table=test-table [id=0ecd8d692ea949289d9e6a68283a71ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:41.777988 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fdfef98416437db0fbc9dfa8f71a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:41.781817 13646 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Bootstrap starting.
I20260619 02:53:41.782033 13648 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: Bootstrap starting.
I20260619 02:53:41.782863 13648 tablet_bootstrap.cc:654] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:41.782915 13646 tablet_bootstrap.cc:654] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:41.783308 13648 log.cc:824] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:41.783334 13646 log.cc:824] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:41.783787 13647 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Bootstrap starting.
I20260619 02:53:41.784258 13646 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: No bootstrap required, opened a new log
I20260619 02:53:41.784330 13646 ts_tablet_manager.cc:1397] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:41.784468 13647 tablet_bootstrap.cc:654] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:41.784792 13647 log.cc:824] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:41.785815 13648 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: No bootstrap required, opened a new log
I20260619 02:53:41.785912 13648 ts_tablet_manager.cc:1397] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 02:53:41.786062 13647 tablet_bootstrap.cc:492] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: No bootstrap required, opened a new log
I20260619 02:53:41.786165 13647 ts_tablet_manager.cc:1397] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:41.787098 13646 raft_consensus.cc:348] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.787364 13646 raft_consensus.cc:374] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:41.787397 13646 raft_consensus.cc:729] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03714d62f224215b670189d11d4b4b9, State: Initialized, Role: FOLLOWER
I20260619 02:53:41.787575 13646 consensus_queue.cc:260] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.787930 13646 ts_tablet_manager.cc:1428] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 02:53:41.788062 13498 heartbeater.cc:503] Master 127.26.98.62:39639 was elected leader, sending a full tablet report...
I20260619 02:53:41.788690 13648 raft_consensus.cc:348] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.788946 13648 raft_consensus.cc:374] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:41.788987 13648 raft_consensus.cc:729] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94b8a28489f45fc85cbf42b477243ec, State: Initialized, Role: FOLLOWER
I20260619 02:53:41.789021 13647 raft_consensus.cc:348] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.789121 13648 consensus_queue.cc:260] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.789196 13647 raft_consensus.cc:374] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:41.789233 13647 raft_consensus.cc:729] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55afa1233e747a288acb71bbe67829f, State: Initialized, Role: FOLLOWER
I20260619 02:53:41.789367 13647 consensus_queue.cc:260] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.789470 13648 ts_tablet_manager.cc:1428] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 02:53:41.789637 13647 ts_tablet_manager.cc:1428] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 02:53:41.789635 13367 heartbeater.cc:503] Master 127.26.98.62:39639 was elected leader, sending a full tablet report...
I20260619 02:53:41.789738 13629 heartbeater.cc:503] Master 127.26.98.62:39639 was elected leader, sending a full tablet report...
W20260619 02:53:41.818593 13499 tablet.cc:2369] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:41.836390 13653 raft_consensus.cc:482] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:41.836572 13653 raft_consensus.cc:504] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.837045 13653 leader_election.cc:288] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b03714d62f224215b670189d11d4b4b9 (127.26.98.2:40689), a55afa1233e747a288acb71bbe67829f (127.26.98.3:45947)
I20260619 02:53:41.840255 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.837371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:48539 (local address 127.26.98.2:40689)
0619 02:53:41.837445 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.837449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.837464 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:53:41.837502 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.837504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.837520 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.837594 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:41.837801 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.838350 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.839466 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.839881 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.839953 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.839968 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.840118 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.840164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:53:41.840260 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.837316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40689 (local address 127.26.98.1:48539)
0619 02:53:41.837368 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.837382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.837439 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.837638 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.837640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.837642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:41.837728 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.837730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.838454 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.838456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.839351 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.839354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.839969 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.839970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.840046 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.840050 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.840100 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.840160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":23}
I20260619 02:53:41.840317 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.837555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:53715 (local address 127.26.98.3:45947)
0619 02:53:41.837607 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.837609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.837744 (+ 135us) server_negotiation.cc:407] Connection header received
0619 02:53:41.837788 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.837789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.837798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.837850 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:41.838002 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.838494 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.839550 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.839962 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.840026 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.840037 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.840191 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.840246 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:53:41.840390 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.837435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:45947 (local address 127.26.98.1:53715)
0619 02:53:41.837667 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.837675 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:41.837733 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:41.837860 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:41.837861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:41.837862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:41.837938 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.837940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.838582 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.838583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.839452 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:41.839454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.840045 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.840046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:41.840126 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.840129 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:41.840176 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:53:41.840228 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":158,"threads_started":1}
I20260619 02:53:41.840760 13583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fdfef98416437db0fbc9dfa8f71a1b" candidate_uuid: "e94b8a28489f45fc85cbf42b477243ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55afa1233e747a288acb71bbe67829f" is_pre_election: true
I20260619 02:53:41.840736 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fdfef98416437db0fbc9dfa8f71a1b" candidate_uuid: "e94b8a28489f45fc85cbf42b477243ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03714d62f224215b670189d11d4b4b9" is_pre_election: true
I20260619 02:53:41.840957 13583 raft_consensus.cc:2454] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94b8a28489f45fc85cbf42b477243ec in term 0.
I20260619 02:53:41.840958 13452 raft_consensus.cc:2454] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94b8a28489f45fc85cbf42b477243ec in term 0.
I20260619 02:53:41.841260 13256 leader_election.cc:302] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55afa1233e747a288acb71bbe67829f, e94b8a28489f45fc85cbf42b477243ec; no voters:
I20260619 02:53:41.841408 13653 raft_consensus.cc:2781] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:41.841468 13653 raft_consensus.cc:482] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:41.841493 13653 raft_consensus.cc:3037] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:41.842374 13653 raft_consensus.cc:504] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.842540 13653 leader_election.cc:288] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [CANDIDATE]: Term 1 election: Requested vote from peers b03714d62f224215b670189d11d4b4b9 (127.26.98.2:40689), a55afa1233e747a288acb71bbe67829f (127.26.98.3:45947)
I20260619 02:53:41.842970 13583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fdfef98416437db0fbc9dfa8f71a1b" candidate_uuid: "e94b8a28489f45fc85cbf42b477243ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55afa1233e747a288acb71bbe67829f"
I20260619 02:53:41.842964 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fdfef98416437db0fbc9dfa8f71a1b" candidate_uuid: "e94b8a28489f45fc85cbf42b477243ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b03714d62f224215b670189d11d4b4b9"
I20260619 02:53:41.843084 13583 raft_consensus.cc:3037] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:41.843107 13452 raft_consensus.cc:3037] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:41.844432 13583 raft_consensus.cc:2454] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94b8a28489f45fc85cbf42b477243ec in term 1.
I20260619 02:53:41.844432 13452 raft_consensus.cc:2454] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94b8a28489f45fc85cbf42b477243ec in term 1.
I20260619 02:53:41.844698 13256 leader_election.cc:302] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55afa1233e747a288acb71bbe67829f, e94b8a28489f45fc85cbf42b477243ec; no voters:
I20260619 02:53:41.844812 13653 raft_consensus.cc:2781] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:41.845008 13653 raft_consensus.cc:686] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [term 1 LEADER]: Becoming Leader. State: Replica: e94b8a28489f45fc85cbf42b477243ec, State: Running, Role: LEADER
I20260619 02:53:41.845134 13653 consensus_queue.cc:237] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } }
I20260619 02:53:41.846372 13179 catalog_manager.cc:5495] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec reported cstate change: term changed from 0 to 1, leader changed from <none> to e94b8a28489f45fc85cbf42b477243ec (127.26.98.1). New cstate: current_term: 1 leader_uuid: "e94b8a28489f45fc85cbf42b477243ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94b8a28489f45fc85cbf42b477243ec" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 } health_report { overall_health: UNKNOWN } } }
I20260619 02:53:41.872546 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.869409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.26.98.1:39625)
0619 02:53:41.869514 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.869516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.869526 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:41.869571 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.869572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.869597 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.869669 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:41.869923 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.870467 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.871483 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.871706 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.871767 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.871998 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:41.872120 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:41.872206 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.872376 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.872393 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:53:41.875175 13452 raft_consensus.cc:1261] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 1 FOLLOWER]: Refusing update from remote peer e94b8a28489f45fc85cbf42b477243ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:41.875204 13583 raft_consensus.cc:1261] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 1 FOLLOWER]: Refusing update from remote peer e94b8a28489f45fc85cbf42b477243ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:41.875741 13656 consensus_queue.cc:1035] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55afa1233e747a288acb71bbe67829f" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:41.875924 13653 consensus_queue.cc:1035] T 98fdfef98416437db0fbc9dfa8f71a1b P e94b8a28489f45fc85cbf42b477243ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03714d62f224215b670189d11d4b4b9" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:41.879671 13660 mvcc.cc:205] Tried to move back new op lower bound from 7298406899196768256 to 7298406899078496256. Current Snapshot: MvccSnapshot[applied={T|T < 7298406899196768256}]
I20260619 02:53:41.883728 13179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:41.884061 13179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:41.885360 13179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 688489de571a4d7eb04e136df30bb499: Sending DeleteTablet for 3 replicas of tablet 98fdfef98416437db0fbc9dfa8f71a1b
I20260619 02:53:41.886132 13432 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:41 UTC) from {username='slave'} at 127.0.0.1:46800
I20260619 02:53:41.886137 13301 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:41 UTC) from {username='slave'} at 127.0.0.1:47944
I20260619 02:53:41.886178 13563 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fdfef98416437db0fbc9dfa8f71a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:41 UTC) from {username='slave'} at 127.0.0.1:40828
I20260619 02:53:41.886703 13669 tablet_replica.cc:330] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: stopping tablet replica
I20260619 02:53:41.886909 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13239
I20260619 02:53:41.886927 13669 raft_consensus.cc:2229] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:41.888648 13671 tablet_replica.cc:330] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: stopping tablet replica
I20260619 02:53:41.888849 13671 raft_consensus.cc:2229] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:41.889015 13671 raft_consensus.cc:2258] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:41.889261 13669 raft_consensus.cc:2258] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:41.889987 13671 ts_tablet_manager.cc:1905] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:41.890004 13669 ts_tablet_manager.cc:1905] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:41.892079 13669 ts_tablet_manager.cc:1918] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:41.892127 13671 ts_tablet_manager.cc:1918] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:41.892201 13669 log.cc:1196] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/98fdfef98416437db0fbc9dfa8f71a1b
I20260619 02:53:41.892249 13671 log.cc:1196] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/98fdfef98416437db0fbc9dfa8f71a1b
I20260619 02:53:41.892544 13671 ts_tablet_manager.cc:1939] T 98fdfef98416437db0fbc9dfa8f71a1b P a55afa1233e747a288acb71bbe67829f: Deleting consensus metadata
I20260619 02:53:41.892590 13669 ts_tablet_manager.cc:1939] T 98fdfef98416437db0fbc9dfa8f71a1b P b03714d62f224215b670189d11d4b4b9: Deleting consensus metadata
I20260619 02:53:41.893260 13165 catalog_manager.cc:4841] TS b03714d62f224215b670189d11d4b4b9 (127.26.98.2:40689): tablet 98fdfef98416437db0fbc9dfa8f71a1b (table test-table [id=0ecd8d692ea949289d9e6a68283a71ee]) successfully deleted
I20260619 02:53:41.893463 13165 catalog_manager.cc:4841] TS a55afa1233e747a288acb71bbe67829f (127.26.98.3:45947): tablet 98fdfef98416437db0fbc9dfa8f71a1b (table test-table [id=0ecd8d692ea949289d9e6a68283a71ee]) successfully deleted
W20260619 02:53:41.894905 13164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:39625 (error 108)
I20260619 02:53:41.895754 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13370
I20260619 02:53:41.895922 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.895645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39625 (local address 127.0.0.1:47966)
0619 02:53:41.895790 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:41.895870 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 02:53:41.896111 13164 catalog_manager.cc:4584] TS e94b8a28489f45fc85cbf42b477243ec (127.26.98.1:39625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98fdfef98416437db0fbc9dfa8f71a1b: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39625: connect: Connection refused (error 111)
I20260619 02:53:41.902182 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13501
I20260619 02:53:41.907647 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13148
2026-06-19T02:53:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (913 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:41 dist-test-slave-0f9d krb5kdc[13675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:41 dist-test-slave-0f9d krb5kdc[13675](info): set up 2 sockets
Jun 19 02:53:41 dist-test-slave-0f9d krb5kdc[13675](info): commencing operation
krb5kdc: starting...
W20260619 02:53:42.911324 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:53:43.055104 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40969
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:43.157961 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:43.158308 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:43.158340 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:43.160982 13688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:43.161147 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:43.161170 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:43.161185 13688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:43.161196 13688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:43.163901 13688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:43.164556 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:43.164935 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:43.171041 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.171049 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.171049 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.171406 13688 server_base.cc:1034] running on GCE node
I20260619 02:53:43.171718 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:43.172119 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:43.173317 13688 hybrid_clock.cc:648] HybridClock initialized: now 1781837623173293 us; error 45 us; skew 500 ppm
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:43.178591 13688 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:53:43.179488 13688 webserver.cc:460] Webserver started at http://127.26.98.62:34625/ using document root <none> and password file <none>
I20260619 02:53:43.179760 13688 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:43.179834 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:43.179979 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:43.181488 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "7596c77b2cc24fef9cf6b14a96733946"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "af545ea06310a51666fecbb5fa1e5598"
server_key_iv: "73fdcda1d5ab9f91de84497a3833bf1a"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.181975 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "7596c77b2cc24fef9cf6b14a96733946"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "af545ea06310a51666fecbb5fa1e5598"
server_key_iv: "73fdcda1d5ab9f91de84497a3833bf1a"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.183811 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260619 02:53:43.184980 13703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.185310 13688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.185428 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "7596c77b2cc24fef9cf6b14a96733946"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "af545ea06310a51666fecbb5fa1e5598"
server_key_iv: "73fdcda1d5ab9f91de84497a3833bf1a"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.185521 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:43.198468 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:43.202046 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:43.202458 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:43.208112 13688 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:40969
I20260619 02:53:43.208153 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:40969 every 8 connection(s)
I20260619 02:53:43.208659 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:43.210047 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:43.212232 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13688
I20260619 02:53:43.212378 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:43.212598 27016 external_mini_cluster.cc:1428] Setting key 857e748a493a8f3c4cd4e19fd0347fb2
I20260619 02:53:43.213541 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: Bootstrap starting.
I20260619 02:53:43.214262 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:43.214591 13756 log.cc:824] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:43.215739 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: No bootstrap required, opened a new log
I20260619 02:53:43.218277 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } }
I20260619 02:53:43.218501 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:43.218538 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7596c77b2cc24fef9cf6b14a96733946, State: Initialized, Role: FOLLOWER
I20260619 02:53:43.218708 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } }
I20260619 02:53:43.218808 13756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:43.218859 13756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:43.218957 13756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:43.220508 13756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } }
I20260619 02:53:43.220709 13756 leader_election.cc:302] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7596c77b2cc24fef9cf6b14a96733946; no voters:
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:43.221222 13756 leader_election.cc:288] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:43.221256 13761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:43.221663 13761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [term 1 LEADER]: Becoming Leader. State: Replica: 7596c77b2cc24fef9cf6b14a96733946, State: Running, Role: LEADER
I20260619 02:53:43.221781 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:43.221853 13761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } }
I20260619 02:53:43.222359 13761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7596c77b2cc24fef9cf6b14a96733946. Latest consensus state: current_term: 1 leader_uuid: "7596c77b2cc24fef9cf6b14a96733946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } } }
I20260619 02:53:43.222474 13761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:43.222460 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7596c77b2cc24fef9cf6b14a96733946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7596c77b2cc24fef9cf6b14a96733946" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 40969 } } }
I20260619 02:53:43.222841 13770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:43.222610 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:43.223606 13770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:43.225849 13770 catalog_manager.cc:1347] Generated new cluster ID: 92b26cb98adb48da9a08f5d5963bbec3
I20260619 02:53:43.225915 13770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:43.226804 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.213360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.26.98.62:40969)
0619 02:53:43.213557 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.213568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.213600 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:53:43.214281 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.214292 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.214381 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.214564 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.215585 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.216651 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.217510 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.218032 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.218162 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.221874 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.221882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.221882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.221901 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.224638 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.225266 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.225270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.225271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.225404 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.225642 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.225643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.225644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.226032 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.226207 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.226393 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.226565 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260619 02:53:43.241374 13770 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:43.242233 13770 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:43.247802 13770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: Generated new TSK 0
I20260619 02:53:43.248111 13770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
Entry for principal kudu/127.26.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab.
I20260619 02:53:43.280128 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:43.380237 13782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:43.380533 13782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:43.380566 13782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:43.382035 13782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:43.382093 13782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:43.382128 13782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:43.384406 13782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.1.keytab
--principal=kudu/127.26.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:43.384769 13782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:43.385036 13782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:43.391243 13787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.391269 13790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.391343 13788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.391938 13782 server_base.cc:1034] running on GCE node
I20260619 02:53:43.392194 13782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:43.392524 13782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:43.393677 13782 hybrid_clock.cc:648] HybridClock initialized: now 1781837623393658 us; error 42 us; skew 500 ppm
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:43.396991 13782 init.cc:377] Logged in from keytab as kudu/127.26.98.1@KRBTEST.COM (short username kudu)
I20260619 02:53:43.397589 13782 webserver.cc:460] Webserver started at http://127.26.98.1:42261/ using document root <none> and password file <none>
I20260619 02:53:43.397843 13782 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:43.397899 13782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:43.398041 13782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:43.399387 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "79f3f5df4dfd405e8aede27a5ecae71c"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "c61dcc653679cd73727b9d776ab686fe"
server_key_iv: "670d7db5e3716a0ff9f714086de9bf6f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.399799 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "79f3f5df4dfd405e8aede27a5ecae71c"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "c61dcc653679cd73727b9d776ab686fe"
server_key_iv: "670d7db5e3716a0ff9f714086de9bf6f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.402096 13782 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:43.403205 13797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.403652 13782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.403789 13782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "79f3f5df4dfd405e8aede27a5ecae71c"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "c61dcc653679cd73727b9d776ab686fe"
server_key_iv: "670d7db5e3716a0ff9f714086de9bf6f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.403887 13782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:43.425604 13782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:43.428228 13782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:43.428658 13782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:43.429028 13782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:43.429484 13782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:43.429525 13782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.429571 13782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:43.429600 13782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.437232 13782 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:39435
I20260619 02:53:43.437500 13910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:39435 every 8 connection(s)
I20260619 02:53:43.437789 13782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.1@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:43.447661 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13782
I20260619 02:53:43.447808 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:43.448063 27016 external_mini_cluster.cc:1428] Setting key ec37e64f1c53e7595851b75d409cacd4
I20260619 02:53:43.450636 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.438806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:34615 (local address 127.26.98.62:40969)
0619 02:53:43.438917 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.438920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.439163 (+ 243us) server_negotiation.cc:407] Connection header received
0619 02:53:43.439828 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.439829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.439845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.439902 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.441082 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.441681 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.442937 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.443194 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.443258 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.446145 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.446148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.446148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.446158 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.448076 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.448997 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.449005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.449006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.449106 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.449619 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.449622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.449622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.449819 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.449913 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.450419 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.450544 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:53:43.450867 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.438750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40969 (local address 127.26.98.1:34615)
0619 02:53:43.439013 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.439061 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.439700 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.439991 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.439997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.440694 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:43.440959 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.440965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.441851 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.441852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.442811 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.442814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.443280 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.443281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.443438 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.444261 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:43.444273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:43.446003 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:43.448268 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.448289 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.448293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.448728 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.449223 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.449226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.449226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.449479 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.449917 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:43.449926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:43.450177 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.450319 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.450569 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":147,"threads_started":1}
I20260619 02:53:43.451632 13911 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40969
I20260619 02:53:43.451773 13911 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:43.452033 13911 heartbeater.cc:511] Master 127.26.98.62:40969 requested a full tablet report, sending...
I20260619 02:53:43.452718 13720 ts_manager.cc:194] Registered new tserver with Master: 79f3f5df4dfd405e8aede27a5ecae71c (127.26.98.1:39435)
I20260619 02:53:43.453550 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.1@KRBTEST.COM'} at 127.26.98.1:34615
WARNING: no policy specified for kudu/127.26.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
Entry for principal kudu/127.26.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab.
I20260619 02:53:43.486838 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:43.592242 13916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:43.592541 13916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:43.592573 13916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:43.594102 13916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:43.594168 13916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:43.594203 13916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:43.596227 13916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.2.keytab
--principal=kudu/127.26.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:43.596524 13916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:43.596788 13916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:43.602756 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.602820 13921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.602756 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.603344 13916 server_base.cc:1034] running on GCE node
I20260619 02:53:43.603615 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:43.603909 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:43.605090 13916 hybrid_clock.cc:648] HybridClock initialized: now 1781837623605070 us; error 42 us; skew 500 ppm
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:43.608237 13916 init.cc:377] Logged in from keytab as kudu/127.26.98.2@KRBTEST.COM (short username kudu)
I20260619 02:53:43.608867 13916 webserver.cc:460] Webserver started at http://127.26.98.2:42467/ using document root <none> and password file <none>
I20260619 02:53:43.609146 13916 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:43.609205 13916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:43.609354 13916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:43.610671 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "eb8819a048b8464380ed9fc1ad79a520"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "7d2355a9bb8c5d7495f10877d8473d82"
server_key_iv: "c191b80a7baafc8bf3eec6886435874f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.611104 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "eb8819a048b8464380ed9fc1ad79a520"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "7d2355a9bb8c5d7495f10877d8473d82"
server_key_iv: "c191b80a7baafc8bf3eec6886435874f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.612651 13916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.613694 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.613926 13916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.614035 13916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "eb8819a048b8464380ed9fc1ad79a520"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "7d2355a9bb8c5d7495f10877d8473d82"
server_key_iv: "c191b80a7baafc8bf3eec6886435874f"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.614110 13916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:43.636195 13916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:43.639313 13916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:43.639489 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:43.639766 13916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:43.640123 13916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:43.640161 13916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.640194 13916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:43.640211 13916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.647879 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:43413 every 8 connection(s)
I20260619 02:53:43.647893 13916 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:43413
I20260619 02:53:43.648495 13916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:43.655560 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 13916
I20260619 02:53:43.655717 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:43.655958 27016 external_mini_cluster.cc:1428] Setting key 57097f8391a6775ebfdb225df26d17a8
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.2@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:43.661923 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.649586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:32871 (local address 127.26.98.62:40969)
0619 02:53:43.649668 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.649670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.649882 (+ 212us) server_negotiation.cc:407] Connection header received
0619 02:53:43.650560 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.650562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.650580 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.650644 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.652002 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.652588 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.653891 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.654108 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.654179 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.657332 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.657334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.657334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.657345 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.659518 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.660168 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.660170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.660170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.660265 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.660846 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.660847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.660848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.661055 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.661154 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.661642 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.661791 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:43.662077 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.649444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40969 (local address 127.26.98.2:32871)
0619 02:53:43.649713 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.649768 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.650426 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.650771 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.650776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.651556 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:43.651852 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.651860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.652771 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.652772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.653778 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.653780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.654204 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.654205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.654357 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.655139 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:43.655150 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:43.657216 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:43.659701 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.659714 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.659717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.660063 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.660471 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.660474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.660474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.660712 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.661174 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:43.661182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:43.661437 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.661570 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.661843 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260619 02:53:43.663002 14045 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40969
I20260619 02:53:43.663146 14045 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:43.663411 14045 heartbeater.cc:511] Master 127.26.98.62:40969 requested a full tablet report, sending...
I20260619 02:53:43.664213 13720 ts_manager.cc:194] Registered new tserver with Master: eb8819a048b8464380ed9fc1ad79a520 (127.26.98.2:43413)
I20260619 02:53:43.664804 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.2@KRBTEST.COM'} at 127.26.98.2:32871
WARNING: no policy specified for kudu/127.26.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
Entry for principal kudu/127.26.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab.
I20260619 02:53:43.688118 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:43.790155 14050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:43.790444 14050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:43.790481 14050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:43.792830 14050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:43.792964 14050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:43.793002 14050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:43.795701 14050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.3.keytab
--principal=kudu/127.26.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:43.796080 14050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:43.796378 14050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:43.802711 14056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.802711 14058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.802713 14055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.803122 14050 server_base.cc:1034] running on GCE node
I20260619 02:53:43.803359 14050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:43.803661 14050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:43.804833 14050 hybrid_clock.cc:648] HybridClock initialized: now 1781837623804784 us; error 63 us; skew 500 ppm
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:43.808333 14050 init.cc:377] Logged in from keytab as kudu/127.26.98.3@KRBTEST.COM (short username kudu)
I20260619 02:53:43.808939 14050 webserver.cc:460] Webserver started at http://127.26.98.3:42157/ using document root <none> and password file <none>
I20260619 02:53:43.809212 14050 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:43.809269 14050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:43.809408 14050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:43.810806 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "e610138316cc4d998c5b84f0d26f0b21"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "f9454a9506a28e453486298e9b70c730"
server_key_iv: "d903b0dd510e96125bd97fac5be117ae"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.811370 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "e610138316cc4d998c5b84f0d26f0b21"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "f9454a9506a28e453486298e9b70c730"
server_key_iv: "d903b0dd510e96125bd97fac5be117ae"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.813015 14050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:43.814282 14065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.814599 14050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.814740 14050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "e610138316cc4d998c5b84f0d26f0b21"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-0f9d"
server_key: "f9454a9506a28e453486298e9b70c730"
server_key_iv: "d903b0dd510e96125bd97fac5be117ae"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.814829 14050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:43.827584 14050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:43.830564 14050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:43.830788 14050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:43.831184 14050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:43.831590 14050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:43.831655 14050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.831696 14050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:43.831713 14050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.839519 14050 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:33213
I20260619 02:53:43.839697 14178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:33213 every 8 connection(s)
I20260619 02:53:43.840039 14050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:43.845525 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14050
I20260619 02:53:43.845664 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:43.845863 27016 external_mini_cluster.cc:1428] Setting key d36f60bf2c88a46f1eac03a4b15aed1a
Jun 19 02:53:43 dist-test-slave-0f9d krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.3@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:53:43.853605 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.841116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:56241 (local address 127.26.98.62:40969)
0619 02:53:43.841231 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.841234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.841401 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:53:43.842122 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.842124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.842141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.842199 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.843593 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.844190 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.845578 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.845858 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.845959 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.849332 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.849334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.849334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.849347 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.851446 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.852240 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.852242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.852243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.852325 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.852768 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.852769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.852770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.852937 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.853005 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.853390 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.853503 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:53:43.853869 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.840983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:40969 (local address 127.26.98.3:56241)
0619 02:53:43.841231 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.841293 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.841986 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.842303 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.842307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.843133 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:43.843419 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.843427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.844305 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.844307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.845393 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.845398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.845980 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.845982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.846175 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.847036 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:43.847049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:43.849158 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:43.851711 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.851729 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.851733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.852101 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.852443 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:43.852445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:43.852446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:43.852619 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:43.853032 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:43.853040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:43.853230 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.853349 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.853598 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260619 02:53:43.854619 14179 heartbeater.cc:347] Connected to a master server at 127.26.98.62:40969
I20260619 02:53:43.854751 14179 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:43.855023 14179 heartbeater.cc:511] Master 127.26.98.62:40969 requested a full tablet report, sending...
I20260619 02:53:43.855710 13720 ts_manager.cc:194] Registered new tserver with Master: e610138316cc4d998c5b84f0d26f0b21 (127.26.98.3:33213)
I20260619 02:53:43.856187 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.98.3@KRBTEST.COM'} at 127.26.98.3:56241
I20260619 02:53:43.859094 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:43.868844 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.860965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.26.98.62:40969)
0619 02:53:43.861035 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.861038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.861102 (+ 64us) server_negotiation.cc:407] Connection header received
0619 02:53:43.861172 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.861173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.861192 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.861254 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.862056 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.862614 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.863617 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.863853 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.863917 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.865355 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.865358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.865358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.865368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.867078 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.867543 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.867544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.867545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.867615 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.867942 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.867944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.867944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.868197 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.868343 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.868645 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.868752 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:53:43.871227 13720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:43.873395 13720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:43.881669 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.877318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.26.98.3:33213)
0619 02:53:43.877547 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.877552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.877565 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:53:43.877615 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.877618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.877637 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.877972 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:43.878066 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.878848 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.880467 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881282 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881358 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.881424 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.881487 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.881556 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260619 02:53:43.881654 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.876912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:33213 (local address 127.0.0.1:46804)
0619 02:53:43.877159 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.877199 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.877283 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.877759 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.877763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.877766 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:43.877894 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.877896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.878973 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.878974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.880360 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.880363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881356 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.881418 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.881421 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.881469 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.881521 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":13,"thread_start_us":96,"threads_started":1}
I20260619 02:53:43.881861 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.877361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.26.98.2:43413)
0619 02:53:43.877583 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.877587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.877761 (+ 174us) server_negotiation.cc:407] Connection header received
0619 02:53:43.877805 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.877807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.877821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.877894 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:43.878093 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.879270 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.880521 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881141 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881215 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.881298 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.881700 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.881767 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260619 02:53:43.881654 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.877195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:43413 (local address 127.0.0.1:44692)
0619 02:53:43.877636 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.877645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.877707 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.877974 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.877974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.877977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:43.878017 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.878018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.879497 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.879498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.880439 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.880442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881233 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.881337 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.881356 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.881432 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.881517 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":74,"threads_started":1}
I20260619 02:53:43.882712 13979 tablet_service.cc:1467] Processing CreateTablet for tablet f19a1078a67546b1877e984d37903c4c (DEFAULT_TABLE table=test-table [id=c7b650579e5a4fc8882925a4e9f74f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:43.882702 14113 tablet_service.cc:1467] Processing CreateTablet for tablet f19a1078a67546b1877e984d37903c4c (DEFAULT_TABLE table=test-table [id=c7b650579e5a4fc8882925a4e9f74f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:43.883184 14113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19a1078a67546b1877e984d37903c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:43.883184 13979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19a1078a67546b1877e984d37903c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:43.883397 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.876917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39435 (local address 127.0.0.1:45148)
0619 02:53:43.877325 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.877338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.877405 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.877774 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.877775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.877778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:43.877888 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.877890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.879450 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.879451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.880551 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.880554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881483 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.881551 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.881555 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.881586 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.881638 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":12,"thread_start_us":116,"threads_started":1}
I20260619 02:53:43.884045 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.877075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.26.98.1:39435)
0619 02:53:43.877469 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.877476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.877488 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:43.877541 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.877543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.877666 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.877968 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:43.878064 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.879270 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.880639 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.881387 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.881709 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.883839 (+ 2130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.883875 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.883947 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":107,"threads_started":1}
I20260619 02:53:43.885165 13845 tablet_service.cc:1467] Processing CreateTablet for tablet f19a1078a67546b1877e984d37903c4c (DEFAULT_TABLE table=test-table [id=c7b650579e5a4fc8882925a4e9f74f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:43.885632 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19a1078a67546b1877e984d37903c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:43.886858 14196 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Bootstrap starting.
I20260619 02:53:43.887508 14197 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Bootstrap starting.
I20260619 02:53:43.887828 14196 tablet_bootstrap.cc:654] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:43.888293 14197 tablet_bootstrap.cc:654] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:43.888535 14196 log.cc:824] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:43.888854 14197 log.cc:824] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:43.889375 14200 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Bootstrap starting.
I20260619 02:53:43.889745 14197 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: No bootstrap required, opened a new log
I20260619 02:53:43.890120 14196 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: No bootstrap required, opened a new log
I20260619 02:53:43.890194 14196 ts_tablet_manager.cc:1397] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:43.890225 14200 tablet_bootstrap.cc:654] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:43.890455 14197 ts_tablet_manager.cc:1397] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:43.890825 14200 log.cc:824] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:43.891901 14200 tablet_bootstrap.cc:492] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: No bootstrap required, opened a new log
I20260619 02:53:43.892014 14200 ts_tablet_manager.cc:1397] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:53:43.892575 14197 raft_consensus.cc:348] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.892853 14197 raft_consensus.cc:374] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:43.892897 14197 raft_consensus.cc:729] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610138316cc4d998c5b84f0d26f0b21, State: Initialized, Role: FOLLOWER
I20260619 02:53:43.893108 14197 consensus_queue.cc:260] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.893208 14196 raft_consensus.cc:348] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.893422 14196 raft_consensus.cc:374] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:43.893476 14196 raft_consensus.cc:729] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8819a048b8464380ed9fc1ad79a520, State: Initialized, Role: FOLLOWER
I20260619 02:53:43.893497 14197 ts_tablet_manager.cc:1428] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:43.893654 14179 heartbeater.cc:503] Master 127.26.98.62:40969 was elected leader, sending a full tablet report...
I20260619 02:53:43.893878 14196 consensus_queue.cc:260] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.894289 14196 ts_tablet_manager.cc:1428] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:53:43.894430 14045 heartbeater.cc:503] Master 127.26.98.62:40969 was elected leader, sending a full tablet report...
I20260619 02:53:43.894503 14200 raft_consensus.cc:348] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.894704 14200 raft_consensus.cc:374] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:43.894752 14200 raft_consensus.cc:729] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f3f5df4dfd405e8aede27a5ecae71c, State: Initialized, Role: FOLLOWER
I20260619 02:53:43.894872 14200 consensus_queue.cc:260] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.895242 14200 ts_tablet_manager.cc:1428] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 02:53:43.895246 13911 heartbeater.cc:503] Master 127.26.98.62:40969 was elected leader, sending a full tablet report...
W20260619 02:53:43.899245 14046 tablet.cc:2369] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:43.915127 14203 raft_consensus.cc:482] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:43.915297 14203 raft_consensus.cc:504] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.915791 14203 leader_election.cc:288] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e610138316cc4d998c5b84f0d26f0b21 (127.26.98.3:33213), 79f3f5df4dfd405e8aede27a5ecae71c (127.26.98.1:39435)
I20260619 02:53:43.919479 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.916054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39435 (local address 127.26.98.2:52399)
0619 02:53:43.916126 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.916142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.916195 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.916398 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.916399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.916402 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:43.916498 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.916501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.917280 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.917283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.918286 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.918291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.919126 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.919127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.919217 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.919222 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.919305 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.919371 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":47}
I20260619 02:53:43.919559 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.916170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:52399 (local address 127.26.98.1:39435)
0619 02:53:43.916220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.916222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.916233 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:43.916269 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.916270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.916285 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.916363 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:43.916574 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.917188 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.918419 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.919014 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.919093 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.919110 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.919334 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.919392 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:53:43.920029 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19a1078a67546b1877e984d37903c4c" candidate_uuid: "eb8819a048b8464380ed9fc1ad79a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" is_pre_election: true
I20260619 02:53:43.920243 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.916055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:33213 (local address 127.26.98.2:45325)
0619 02:53:43.916319 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.916327 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:43.916387 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:43.916549 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:43.916550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:43.916552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:43.916630 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.916632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.917286 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.917287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.918514 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:43.918523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.919496 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.919498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:43.920013 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.920023 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:53:43.920093 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:53:43.920155 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":28,"thread_start_us":146,"threads_started":1}
I20260619 02:53:43.920256 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.916225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:45325 (local address 127.26.98.3:33213)
0619 02:53:43.916294 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.916297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.916390 (+ 93us) server_negotiation.cc:407] Connection header received
0619 02:53:43.916443 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.916444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.916458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.916547 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:43.916696 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.917212 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.918673 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.919321 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.919395 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.919409 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.920104 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.920156 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:43.920303 13865 raft_consensus.cc:2454] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8819a048b8464380ed9fc1ad79a520 in term 0.
I20260619 02:53:43.920615 14133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19a1078a67546b1877e984d37903c4c" candidate_uuid: "eb8819a048b8464380ed9fc1ad79a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610138316cc4d998c5b84f0d26f0b21" is_pre_election: true
I20260619 02:53:43.920672 13932 leader_election.cc:302] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f3f5df4dfd405e8aede27a5ecae71c, eb8819a048b8464380ed9fc1ad79a520; no voters:
I20260619 02:53:43.920792 14133 raft_consensus.cc:2454] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8819a048b8464380ed9fc1ad79a520 in term 0.
I20260619 02:53:43.920895 14203 raft_consensus.cc:2781] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:43.920979 14203 raft_consensus.cc:482] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:43.921007 14203 raft_consensus.cc:3037] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:43.922001 14203 raft_consensus.cc:504] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.922299 14203 leader_election.cc:288] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [CANDIDATE]: Term 1 election: Requested vote from peers e610138316cc4d998c5b84f0d26f0b21 (127.26.98.3:33213), 79f3f5df4dfd405e8aede27a5ecae71c (127.26.98.1:39435)
I20260619 02:53:43.922477 14133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19a1078a67546b1877e984d37903c4c" candidate_uuid: "eb8819a048b8464380ed9fc1ad79a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610138316cc4d998c5b84f0d26f0b21"
I20260619 02:53:43.922483 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19a1078a67546b1877e984d37903c4c" candidate_uuid: "eb8819a048b8464380ed9fc1ad79a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f3f5df4dfd405e8aede27a5ecae71c"
I20260619 02:53:43.922612 14133 raft_consensus.cc:3037] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:43.922619 13865 raft_consensus.cc:3037] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:43.923712 14133 raft_consensus.cc:2454] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8819a048b8464380ed9fc1ad79a520 in term 1.
I20260619 02:53:43.923710 13865 raft_consensus.cc:2454] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8819a048b8464380ed9fc1ad79a520 in term 1.
I20260619 02:53:43.923938 13932 leader_election.cc:302] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f3f5df4dfd405e8aede27a5ecae71c, eb8819a048b8464380ed9fc1ad79a520; no voters:
I20260619 02:53:43.924074 14203 raft_consensus.cc:2781] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:43.924320 14203 raft_consensus.cc:686] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 1 LEADER]: Becoming Leader. State: Replica: eb8819a048b8464380ed9fc1ad79a520, State: Running, Role: LEADER
I20260619 02:53:43.924432 14203 consensus_queue.cc:237] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } }
I20260619 02:53:43.925535 13720 catalog_manager.cc:5495] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb8819a048b8464380ed9fc1ad79a520 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "eb8819a048b8464380ed9fc1ad79a520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb8819a048b8464380ed9fc1ad79a520" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 43413 } health_report { overall_health: HEALTHY } } }
I20260619 02:53:43.934340 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.931855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.26.98.2:43413)
0619 02:53:43.931914 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.931916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.931924 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:43.931966 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.931967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.931999 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.932080 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:43.932243 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.932775 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.933610 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.933825 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.933885 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.933996 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:43.934136 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:43.934202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.934215 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.934237 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:53:43.936854 13865 raft_consensus.cc:1261] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c [term 1 FOLLOWER]: Refusing update from remote peer eb8819a048b8464380ed9fc1ad79a520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:43.936854 14133 raft_consensus.cc:1261] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 1 FOLLOWER]: Refusing update from remote peer eb8819a048b8464380ed9fc1ad79a520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:43.937369 14203 consensus_queue.cc:1035] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f3f5df4dfd405e8aede27a5ecae71c" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:43.937647 14206 consensus_queue.cc:1035] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e610138316cc4d998c5b84f0d26f0b21" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:53:43.938774 13912 tablet.cc:2369] T f19a1078a67546b1877e984d37903c4c P 79f3f5df4dfd405e8aede27a5ecae71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:43.941460 14210 mvcc.cc:205] Tried to move back new op lower bound from 7298406907641356288 to 7298406907595300864. Current Snapshot: MvccSnapshot[applied={T|T < 7298406907641356288}]
I20260619 02:53:43.941987 14211 mvcc.cc:205] Tried to move back new op lower bound from 7298406907641356288 to 7298406907595300864. Current Snapshot: MvccSnapshot[applied={T|T < 7298406907641356288}]
I20260619 02:53:43.943921 13719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:43.944051 13719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:43.945235 13719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7596c77b2cc24fef9cf6b14a96733946: Sending DeleteTablet for 3 replicas of tablet f19a1078a67546b1877e984d37903c4c
I20260619 02:53:43.945894 14113 tablet_service.cc:1514] Processing DeleteTablet for tablet f19a1078a67546b1877e984d37903c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:43 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:46804
I20260619 02:53:43.945895 13845 tablet_service.cc:1514] Processing DeleteTablet for tablet f19a1078a67546b1877e984d37903c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:43 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260619 02:53:43.946235 13979 tablet_service.cc:1514] Processing DeleteTablet for tablet f19a1078a67546b1877e984d37903c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:43 UTC) from {username='kudu', principal='kudu/127.26.98.62@KRBTEST.COM'} at 127.0.0.1:44692
I20260619 02:53:43.946480 14221 tablet_replica.cc:330] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: stopping tablet replica
I20260619 02:53:43.946686 14221 raft_consensus.cc:2229] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:43.946692 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13782
I20260619 02:53:43.946995 14221 raft_consensus.cc:2258] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:43.947559 14219 tablet_replica.cc:330] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: stopping tablet replica
I20260619 02:53:43.947710 14219 raft_consensus.cc:2229] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:43.947839 14219 raft_consensus.cc:2258] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:43.948398 14219 ts_tablet_manager.cc:1905] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:43.950647 14219 ts_tablet_manager.cc:1918] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:43.950855 14219 log.cc:1196] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/f19a1078a67546b1877e984d37903c4c
I20260619 02:53:43.951340 14219 ts_tablet_manager.cc:1939] T f19a1078a67546b1877e984d37903c4c P e610138316cc4d998c5b84f0d26f0b21: Deleting consensus metadata
I20260619 02:53:43.951321 14221 ts_tablet_manager.cc:1905] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:43.952495 13707 catalog_manager.cc:4841] TS e610138316cc4d998c5b84f0d26f0b21 (127.26.98.3:33213): tablet f19a1078a67546b1877e984d37903c4c (table test-table [id=c7b650579e5a4fc8882925a4e9f74f02]) successfully deleted
I20260619 02:53:43.953195 14221 ts_tablet_manager.cc:1918] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:43.953291 14221 log.cc:1196] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/f19a1078a67546b1877e984d37903c4c
I20260619 02:53:43.953604 14221 ts_tablet_manager.cc:1939] T f19a1078a67546b1877e984d37903c4c P eb8819a048b8464380ed9fc1ad79a520: Deleting consensus metadata
W20260619 02:53:43.954078 13704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:39435 (error 108)
I20260619 02:53:43.954233 13707 catalog_manager.cc:4841] TS eb8819a048b8464380ed9fc1ad79a520 (127.26.98.2:43413): tablet f19a1078a67546b1877e984d37903c4c (table test-table [id=c7b650579e5a4fc8882925a4e9f74f02]) successfully deleted
I20260619 02:53:43.954803 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.954634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:39435 (local address 127.0.0.1:45158)
0619 02:53:43.954705 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:43.954760 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 02:53:43.954851 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13916
W20260619 02:53:43.956765 13704 catalog_manager.cc:4584] TS 79f3f5df4dfd405e8aede27a5ecae71c (127.26.98.1:39435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f19a1078a67546b1877e984d37903c4c: Network error: Client connection negotiation failed: client connection to 127.26.98.1:39435: connect: Connection refused (error 111)
I20260619 02:53:43.961234 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14050
I20260619 02:53:43.966594 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 13688
2026-06-19T02:53:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2060 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T02:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:43Z Disabled control of system clock
I20260619 02:53:43.999558 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43203
--webserver_interface=127.26.98.62
--webserver_port=0
--builtin_ntp_servers=127.26.98.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:43203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:44.100366 14226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:44.100677 14226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:44.100713 14226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:44.102312 14226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:44.102396 14226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:44.102412 14226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:44.102429 14226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:44.102445 14226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:44.104806 14226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:43203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:43203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:44.105238 14226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:44.105639 14226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:44.112259 14231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.112260 14232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.112260 14234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.112586 14226 server_base.cc:1034] running on GCE node
I20260619 02:53:44.112862 14226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:44.113243 14226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:44.114499 14226 hybrid_clock.cc:648] HybridClock initialized: now 1781837624114426 us; error 50 us; skew 500 ppm
I20260619 02:53:44.116268 14226 webserver.cc:460] Webserver started at http://127.26.98.62:36301/ using document root <none> and password file <none>
I20260619 02:53:44.116602 14226 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:44.116658 14226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:44.116895 14226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:44.118265 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "bf506851ec48410a90929213fb4d36d3"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "9fd75401aa18b4282c7e52effc657b7b"
server_key_iv: "f48ef75782141391a38d4ee536e626a3"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.118712 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "bf506851ec48410a90929213fb4d36d3"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "9fd75401aa18b4282c7e52effc657b7b"
server_key_iv: "f48ef75782141391a38d4ee536e626a3"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.120378 14226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:44.121721 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.122195 14226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:44.122334 14226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "bf506851ec48410a90929213fb4d36d3"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "9fd75401aa18b4282c7e52effc657b7b"
server_key_iv: "f48ef75782141391a38d4ee536e626a3"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.122450 14226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:44.137967 14226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:44.138408 14226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:44.138577 14226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:44.143950 14226 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:43203
I20260619 02:53:44.144018 14292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:43203 every 8 connection(s)
I20260619 02:53:44.144465 14226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:44.145711 14293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:44.147127 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14226
I20260619 02:53:44.147240 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:44.147454 27016 external_mini_cluster.cc:1428] Setting key b5fd7e2b80329e02065478c5d64f5151
I20260619 02:53:44.149549 14293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: Bootstrap starting.
I20260619 02:53:44.150467 14293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:44.150903 14293 log.cc:824] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:44.152170 14293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: No bootstrap required, opened a new log
I20260619 02:53:44.154618 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.148220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.26.98.62:43203)
0619 02:53:44.148546 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.148558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.148598 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:53:44.149211 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.149222 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.149326 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.149535 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:44.149910 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.151387 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.152389 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.152867 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.152992 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.153862 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:44.153868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:44.153868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:44.154076 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:44.154116 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:44.154132 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:44.154191 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.154284 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.154391 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260619 02:53:44.154645 14293 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } }
I20260619 02:53:44.154852 14293 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:44.154929 14293 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf506851ec48410a90929213fb4d36d3, State: Initialized, Role: FOLLOWER
I20260619 02:53:44.155078 14293 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } }
I20260619 02:53:44.155158 14293 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:44.155190 14293 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:44.155241 14293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:44.156031 14293 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } }
I20260619 02:53:44.156159 14293 leader_election.cc:302] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf506851ec48410a90929213fb4d36d3; no voters:
I20260619 02:53:44.156386 14293 leader_election.cc:288] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:44.156476 14298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:44.156656 14298 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [term 1 LEADER]: Becoming Leader. State: Replica: bf506851ec48410a90929213fb4d36d3, State: Running, Role: LEADER
I20260619 02:53:44.156741 14293 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:44.156771 14298 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } }
I20260619 02:53:44.158159 14299 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf506851ec48410a90929213fb4d36d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } } }
I20260619 02:53:44.158387 14299 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:44.158643 14300 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf506851ec48410a90929213fb4d36d3. Latest consensus state: current_term: 1 leader_uuid: "bf506851ec48410a90929213fb4d36d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf506851ec48410a90929213fb4d36d3" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 43203 } } }
I20260619 02:53:44.158814 14300 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:44.159175 14307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:44.159838 14307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:44.162287 14307 catalog_manager.cc:1347] Generated new cluster ID: 9de900cfda1647b6a806a6918cc0b5c7
I20260619 02:53:44.162549 14307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:44.170074 14307 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:44.171060 14307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:44.175211 14307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: Generated new TSK 0
I20260619 02:53:44.175524 14307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:44.181602 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.1:0
--local_ip_for_outbound_sockets=127.26.98.1
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--builtin_ntp_servers=127.26.98.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:44.294265 14317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:44.294588 14317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:44.294628 14317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:44.296267 14317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:44.296358 14317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:44.296406 14317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.1
I20260619 02:53:44.298275 14317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.98.1
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.1
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:44.298640 14317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:44.299072 14317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:44.307327 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.307327 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.307511 14317 server_base.cc:1034] running on GCE node
W20260619 02:53:44.307364 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.307972 14317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:44.308311 14317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:44.309510 14317 hybrid_clock.cc:648] HybridClock initialized: now 1781837624309482 us; error 46 us; skew 500 ppm
I20260619 02:53:44.311269 14317 webserver.cc:460] Webserver started at http://127.26.98.1:34073/ using document root <none> and password file <none>
I20260619 02:53:44.311582 14317 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:44.311642 14317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:44.311784 14317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:44.313115 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data/instance:
uuid: "35f5840cb97d49178bcc2e57718bd0fe"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "7ef6f64e04b996f5d6c78de05cec44d2"
server_key_iv: "19445f80b252c8aae72f168def436ee9"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.313539 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance:
uuid: "35f5840cb97d49178bcc2e57718bd0fe"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "7ef6f64e04b996f5d6c78de05cec44d2"
server_key_iv: "19445f80b252c8aae72f168def436ee9"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.315048 14317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:44.315966 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.316133 14317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:44.316221 14317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal
uuid: "35f5840cb97d49178bcc2e57718bd0fe"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "7ef6f64e04b996f5d6c78de05cec44d2"
server_key_iv: "19445f80b252c8aae72f168def436ee9"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.316310 14317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:44.328053 14317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:44.328464 14317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:44.328610 14317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:44.328907 14317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:44.329326 14317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:44.329365 14317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.329401 14317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:44.329423 14317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.336705 14317 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.1:46475
I20260619 02:53:44.336756 14444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.1:46475 every 8 connection(s)
I20260619 02:53:44.337188 14317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:44.340409 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14317
I20260619 02:53:44.340520 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:44.340771 27016 external_mini_cluster.cc:1428] Setting key 54dcdc642e93bcdffceda7ca76c66ef8
I20260619 02:53:44.341828 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.2:0
--local_ip_for_outbound_sockets=127.26.98.2
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--builtin_ntp_servers=127.26.98.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:44.344045 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.338158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.1:47427 (local address 127.26.98.62:43203)
0619 02:53:44.338284 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.338286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.338485 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:53:44.339207 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.339208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.339230 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.339293 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:44.340194 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.340816 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.342399 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.342623 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.342688 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.343744 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:44.343745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:44.343746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:44.343752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:44.343778 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:44.343783 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:44.343826 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.343908 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.343943 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:53:44.344297 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.338037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43203 (local address 127.26.98.1:47427)
0619 02:53:44.338324 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.338380 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.339055 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.339395 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.339401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.339425 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:44.340070 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.340075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.340921 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.340922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.342230 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.342240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.342716 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.342717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.342949 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.343586 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:44.343586 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:44.343620 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:44.343648 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:44.343833 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:44.343839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:44.343842 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.343908 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.344038 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
I20260619 02:53:44.345063 14445 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43203
I20260619 02:53:44.345242 14445 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:44.345538 14445 heartbeater.cc:511] Master 127.26.98.62:43203 requested a full tablet report, sending...
I20260619 02:53:44.346333 14257 ts_manager.cc:194] Registered new tserver with Master: 35f5840cb97d49178bcc2e57718bd0fe (127.26.98.1:46475)
I20260619 02:53:44.347150 14257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.1:47427
W20260619 02:53:44.439541 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:44.439826 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:44.439864 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:44.441334 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:44.441403 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:44.441447 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.2
I20260619 02:53:44.443260 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.98.2
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.2
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:44.443614 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:44.443997 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:44.450284 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.450284 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.450516 14448 server_base.cc:1034] running on GCE node
W20260619 02:53:44.450326 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.450973 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:44.451390 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:44.452592 14448 hybrid_clock.cc:648] HybridClock initialized: now 1781837624452550 us; error 59 us; skew 500 ppm
I20260619 02:53:44.454487 14448 webserver.cc:460] Webserver started at http://127.26.98.2:35229/ using document root <none> and password file <none>
I20260619 02:53:44.454963 14448 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:44.455107 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:44.455281 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:44.456663 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data/instance:
uuid: "f203f484a2184864830664ae27717885"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "e8f9a8c71eae6f2f0a5948497bea6079"
server_key_iv: "7f564cd203c9986517359985b30af3d7"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.457118 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance:
uuid: "f203f484a2184864830664ae27717885"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "e8f9a8c71eae6f2f0a5948497bea6079"
server_key_iv: "7f564cd203c9986517359985b30af3d7"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.459326 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:53:44.460649 14462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.460955 14448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:44.461061 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal
uuid: "f203f484a2184864830664ae27717885"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "e8f9a8c71eae6f2f0a5948497bea6079"
server_key_iv: "7f564cd203c9986517359985b30af3d7"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.461148 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:44.482024 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:44.482607 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:44.482810 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:44.483114 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:44.483508 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:44.483548 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.483580 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:44.483596 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.491349 14448 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.2:40535
I20260619 02:53:44.491418 14575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.2:40535 every 8 connection(s)
I20260619 02:53:44.491840 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:44.499109 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.492869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:44041 (local address 127.26.98.62:43203)
0619 02:53:44.492966 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.492969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.493183 (+ 214us) server_negotiation.cc:407] Connection header received
0619 02:53:44.493820 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.493821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.493835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.493901 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:44.494807 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.495617 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.496954 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.497185 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.497267 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.498656 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:44.498659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:44.498659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:44.498666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:44.498687 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:44.498692 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:44.498761 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.498964 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.499006 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:53:44.499367 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.492804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43203 (local address 127.26.98.2:44041)
0619 02:53:44.493016 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.493060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.493679 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.493984 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.493988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.493999 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:44.494613 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.494619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.495778 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.495780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.496783 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.496786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.497326 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.497327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.497650 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.498400 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:44.498400 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:44.498442 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:44.498479 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:44.498805 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:44.498812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:44.498816 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.498960 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.499105 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260619 02:53:44.500371 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14448
I20260619 02:53:44.500553 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:44.500746 14576 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43203
I20260619 02:53:44.500836 27016 external_mini_cluster.cc:1428] Setting key c2d382ed348445052073626351c04a53
I20260619 02:53:44.500897 14576 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:44.501268 14576 heartbeater.cc:511] Master 127.26.98.62:43203 requested a full tablet report, sending...
I20260619 02:53:44.501801 14257 ts_manager.cc:194] Registered new tserver with Master: f203f484a2184864830664ae27717885 (127.26.98.2:40535)
I20260619 02:53:44.502190 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.98.3:0
--local_ip_for_outbound_sockets=127.26.98.3
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--builtin_ntp_servers=127.26.98.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:44.502378 14257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.2:44041
W20260619 02:53:44.601109 14579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:44.601428 14579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:44.601459 14579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:44.603490 14579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:44.603588 14579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:44.603626 14579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.98.3
I20260619 02:53:44.605556 14579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.98.3
--webserver_port=0
--tserver_master_addrs=127.26.98.62:43203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.98.3
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:44.605932 14579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:44.606339 14579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:44.612993 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.612993 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.613029 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.613220 14579 server_base.cc:1034] running on GCE node
I20260619 02:53:44.613485 14579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:44.613870 14579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:44.615001 14579 hybrid_clock.cc:648] HybridClock initialized: now 1781837624614983 us; error 32 us; skew 500 ppm
I20260619 02:53:44.616761 14579 webserver.cc:460] Webserver started at http://127.26.98.3:40775/ using document root <none> and password file <none>
I20260619 02:53:44.617057 14579 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:44.617117 14579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:44.617256 14579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:44.618579 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data/instance:
uuid: "5c96146cc4224c63901e8ee430a41907"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "131d706dffa2e92238fabfeedc8e07a1"
server_key_iv: "ff0edf59182a758732d8b353e0ae4019"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.619033 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance:
uuid: "5c96146cc4224c63901e8ee430a41907"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "131d706dffa2e92238fabfeedc8e07a1"
server_key_iv: "ff0edf59182a758732d8b353e0ae4019"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.620548 14579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:44.621421 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.621605 14579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:44.621721 14579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal
uuid: "5c96146cc4224c63901e8ee430a41907"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-0f9d"
server_key: "131d706dffa2e92238fabfeedc8e07a1"
server_key_iv: "ff0edf59182a758732d8b353e0ae4019"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.621783 14579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:44.642722 14579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:44.643213 14579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:44.643358 14579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:44.643633 14579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:44.643988 14579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:44.644024 14579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.644052 14579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:44.644062 14579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.650697 14579 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.3:37267
I20260619 02:53:44.650779 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.3:37267 every 8 connection(s)
I20260619 02:53:44.651185 14579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:44.657809 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.652093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.3:59841 (local address 127.26.98.62:43203)
0619 02:53:44.652176 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.652179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.652411 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:53:44.653100 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.653103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.653121 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.653215 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:44.654163 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.654770 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.655912 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.656120 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.656202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.657367 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:44.657369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:44.657370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:44.657383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:44.657408 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:44.657414 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:44.657474 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.657612 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.657676 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:53:44.658308 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.652052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:43203 (local address 127.26.98.3:59841)
0619 02:53:44.652272 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.652316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.652912 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.653343 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.653348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.653359 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:44.654011 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.654016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.654853 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.654854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.655814 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.655817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.656224 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.656224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.656369 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.657088 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:44.657089 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:44.657137 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:44.657181 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:44.657498 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:44.657502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:44.657506 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.657574 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.657745 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260619 02:53:44.658942 14707 heartbeater.cc:347] Connected to a master server at 127.26.98.62:43203
I20260619 02:53:44.659152 14707 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:44.659173 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14579
I20260619 02:53:44.659267 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:44.659438 14707 heartbeater.cc:511] Master 127.26.98.62:43203 requested a full tablet report, sending...
I20260619 02:53:44.659514 27016 external_mini_cluster.cc:1428] Setting key 39375a47d588c30812d095c4f6a42d8b
I20260619 02:53:44.659941 14256 ts_manager.cc:194] Registered new tserver with Master: 5c96146cc4224c63901e8ee430a41907 (127.26.98.3:37267)
I20260619 02:53:44.660418 14256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.98.3:59841
I20260619 02:53:44.661041 27016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:44.666182 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.662949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.26.98.62:43203)
0619 02:53:44.663013 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.663015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.663088 (+ 73us) server_negotiation.cc:407] Connection header received
0619 02:53:44.663175 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.663176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.663188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.663241 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:44.663408 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.663940 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.664839 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.665056 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.665132 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.665902 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:44.665903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:44.665904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:44.665912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:44.665929 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:44.665934 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:44.665974 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.666028 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.666066 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:53:44.668598 14256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:44.670778 14256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:44.678949 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.673956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:46475 (local address 127.0.0.1:41192)
0619 02:53:44.674190 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.674242 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.674339 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.674756 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.674760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.674763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:44.674861 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.674863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676412 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.676413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.677535 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.677540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.678535 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.678536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.678644 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.678660 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.678733 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.678806 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":140,"thread_start_us":88,"threads_started":1}
I20260619 02:53:44.678961 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.674076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.26.98.1:46475)
0619 02:53:44.674507 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.674513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.674524 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:44.674574 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.674577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.674594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.674666 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:44.674969 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676322 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.677679 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.678433 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.678527 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.678656 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.678747 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.678796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":117,"threads_started":1}
I20260619 02:53:44.679327 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.673960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37267 (local address 127.0.0.1:42984)
0619 02:53:44.674282 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.674291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.674339 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.674873 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.674899 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.674902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:44.674944 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.674946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676347 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.676348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.677845 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.677846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.679070 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.679071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.679146 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.679152 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.679183 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.679239 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":47,"spinlock_wait_cycles":896,"thread_start_us":120,"threads_started":1}
I20260619 02:53:44.679487 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.673960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.2:40535 (local address 127.0.0.1:55220)
0619 02:53:44.674442 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.674451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.674493 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.674851 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.674852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.674856 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:44.674944 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.674948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676433 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.676435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.677824 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.677828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.679206 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.679207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.679258 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.679263 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.679299 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.679338 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":99,"thread_start_us":119,"threads_started":1}
I20260619 02:53:44.679898 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.674341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.26.98.2:40535)
0619 02:53:44.674593 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.674597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.674609 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:44.674657 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.674660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.674676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.674772 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:44.675023 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676246 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.677952 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.678953 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.679355 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.679642 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.679743 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.679800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260619 02:53:44.679899 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.674341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.26.98.3:37267)
0619 02:53:44.674605 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.674609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.674618 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:44.674758 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.674760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.674775 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.675028 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:44.675098 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.676246 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.677952 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.678954 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.679612 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.679721 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.679747 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.679800 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260619 02:53:44.680591 14379 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab (DEFAULT_TABLE table=test-table [id=3d0d13e58bd94049b807d537d033ea5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:44.680790 14510 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab (DEFAULT_TABLE table=test-table [id=3d0d13e58bd94049b807d537d033ea5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:44.680836 14641 tablet_service.cc:1467] Processing CreateTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab (DEFAULT_TABLE table=test-table [id=3d0d13e58bd94049b807d537d033ea5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:44.681155 14379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75808d58db43be9c0e12fe58a8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:44.681195 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75808d58db43be9c0e12fe58a8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:44.681363 14641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d75808d58db43be9c0e12fe58a8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:44.684908 14722 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Bootstrap starting.
I20260619 02:53:44.685138 14721 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Bootstrap starting.
I20260619 02:53:44.685799 14722 tablet_bootstrap.cc:654] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:44.685813 14721 tablet_bootstrap.cc:654] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:44.686219 14721 log.cc:824] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:44.686219 14722 log.cc:824] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:44.686650 14723 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Bootstrap starting.
I20260619 02:53:44.687284 14722 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: No bootstrap required, opened a new log
I20260619 02:53:44.687369 14722 ts_tablet_manager.cc:1397] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:44.687556 14723 tablet_bootstrap.cc:654] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:44.687870 14721 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: No bootstrap required, opened a new log
I20260619 02:53:44.687948 14721 ts_tablet_manager.cc:1397] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:44.687945 14723 log.cc:824] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:44.688943 14723 tablet_bootstrap.cc:492] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: No bootstrap required, opened a new log
I20260619 02:53:44.689020 14723 ts_tablet_manager.cc:1397] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:44.689870 14722 raft_consensus.cc:348] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.690060 14722 raft_consensus.cc:374] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:44.690095 14722 raft_consensus.cc:729] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c96146cc4224c63901e8ee430a41907, State: Initialized, Role: FOLLOWER
I20260619 02:53:44.690243 14722 consensus_queue.cc:260] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.690568 14722 ts_tablet_manager.cc:1428] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:53:44.690599 14707 heartbeater.cc:503] Master 127.26.98.62:43203 was elected leader, sending a full tablet report...
I20260619 02:53:44.691179 14721 raft_consensus.cc:348] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.691377 14721 raft_consensus.cc:374] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:44.691421 14721 raft_consensus.cc:729] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f203f484a2184864830664ae27717885, State: Initialized, Role: FOLLOWER
I20260619 02:53:44.691542 14721 consensus_queue.cc:260] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.691992 14721 ts_tablet_manager.cc:1428] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:53:44.691911 14723 raft_consensus.cc:348] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.692065 14723 raft_consensus.cc:374] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:44.692094 14723 raft_consensus.cc:729] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f5840cb97d49178bcc2e57718bd0fe, State: Initialized, Role: FOLLOWER
I20260619 02:53:44.692107 14576 heartbeater.cc:503] Master 127.26.98.62:43203 was elected leader, sending a full tablet report...
I20260619 02:53:44.692191 14723 consensus_queue.cc:260] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.692476 14723 ts_tablet_manager.cc:1428] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:44.692651 14445 heartbeater.cc:503] Master 127.26.98.62:43203 was elected leader, sending a full tablet report...
W20260619 02:53:44.742612 14577 tablet.cc:2369] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:53:44.838176 14446 tablet.cc:2369] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:44.879623 14728 raft_consensus.cc:482] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:44.879819 14728 raft_consensus.cc:504] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.880290 14728 leader_election.cc:288] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c96146cc4224c63901e8ee430a41907 (127.26.98.3:37267), 35f5840cb97d49178bcc2e57718bd0fe (127.26.98.1:46475)
I20260619 02:53:44.885152 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.880766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:59741 (local address 127.26.98.3:37267)
0619 02:53:44.880840 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.880842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.880882 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:53:44.880919 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.880920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.880930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.881040 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:44.881113 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.881930 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.883740 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.884365 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.884453 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.884467 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.884967 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.885051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:53:44.885152 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.880718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.98.2:56403 (local address 127.26.98.1:46475)
0619 02:53:44.880781 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.880784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.880803 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:53:44.880848 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.880848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.880862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.880942 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:44.881150 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.881929 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.883761 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.884364 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.884452 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.884467 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.884967 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.885056 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:44.885130 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.880551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:46475 (local address 127.26.98.2:56403)
0619 02:53:44.880612 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.880626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.880700 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.880959 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.880960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.880963 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:44.881060 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.881063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.882085 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.882085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.883582 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.883587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.884576 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.884579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.884763 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.884773 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.884899 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.885043 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":103}
I20260619 02:53:44.885140 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.880577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.3:37267 (local address 127.26.98.2:59741)
0619 02:53:44.880818 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.880826 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:44.880874 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:44.880997 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:44.880998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:44.880999 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:44.881060 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.881063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.882081 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.882083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.883582 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:44.883587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.884576 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.884579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:44.884763 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.884774 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:53:44.884899 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:53:44.884995 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":109,"thread_start_us":172,"threads_started":1}
I20260619 02:53:44.885746 14399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75808d58db43be9c0e12fe58a8f4ab" candidate_uuid: "f203f484a2184864830664ae27717885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f5840cb97d49178bcc2e57718bd0fe" is_pre_election: true
I20260619 02:53:44.885865 14661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75808d58db43be9c0e12fe58a8f4ab" candidate_uuid: "f203f484a2184864830664ae27717885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c96146cc4224c63901e8ee430a41907" is_pre_election: true
I20260619 02:53:44.885975 14399 raft_consensus.cc:2454] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f203f484a2184864830664ae27717885 in term 0.
I20260619 02:53:44.886078 14661 raft_consensus.cc:2454] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f203f484a2184864830664ae27717885 in term 0.
I20260619 02:53:44.886312 14465 leader_election.cc:302] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5840cb97d49178bcc2e57718bd0fe, f203f484a2184864830664ae27717885; no voters:
I20260619 02:53:44.886468 14728 raft_consensus.cc:2781] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:44.886540 14728 raft_consensus.cc:482] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:44.886557 14728 raft_consensus.cc:3037] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:44.887617 14728 raft_consensus.cc:504] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.887830 14728 leader_election.cc:288] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [CANDIDATE]: Term 1 election: Requested vote from peers 5c96146cc4224c63901e8ee430a41907 (127.26.98.3:37267), 35f5840cb97d49178bcc2e57718bd0fe (127.26.98.1:46475)
I20260619 02:53:44.888090 14661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75808d58db43be9c0e12fe58a8f4ab" candidate_uuid: "f203f484a2184864830664ae27717885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c96146cc4224c63901e8ee430a41907"
I20260619 02:53:44.888110 14399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d75808d58db43be9c0e12fe58a8f4ab" candidate_uuid: "f203f484a2184864830664ae27717885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f5840cb97d49178bcc2e57718bd0fe"
I20260619 02:53:44.888195 14661 raft_consensus.cc:3037] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:44.888238 14399 raft_consensus.cc:3037] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:44.889181 14399 raft_consensus.cc:2454] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f203f484a2184864830664ae27717885 in term 1.
I20260619 02:53:44.889266 14661 raft_consensus.cc:2454] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f203f484a2184864830664ae27717885 in term 1.
I20260619 02:53:44.889437 14465 leader_election.cc:302] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5840cb97d49178bcc2e57718bd0fe, f203f484a2184864830664ae27717885; no voters:
I20260619 02:53:44.889567 14728 raft_consensus.cc:2781] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:44.889812 14728 raft_consensus.cc:686] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 1 LEADER]: Becoming Leader. State: Replica: f203f484a2184864830664ae27717885, State: Running, Role: LEADER
I20260619 02:53:44.889974 14728 consensus_queue.cc:237] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } }
I20260619 02:53:44.891191 14256 catalog_manager.cc:5495] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 reported cstate change: term changed from 0 to 1, leader changed from <none> to f203f484a2184864830664ae27717885 (127.26.98.2). New cstate: current_term: 1 leader_uuid: "f203f484a2184864830664ae27717885" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f203f484a2184864830664ae27717885" member_type: VOTER last_known_addr { host: "127.26.98.2" port: 40535 } health_report { overall_health: HEALTHY } } }
W20260619 02:53:44.901875 14708 tablet.cc:2369] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:44.910804 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.908122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.26.98.2:40535)
0619 02:53:44.908211 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:44.908213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:44.908223 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:44.908315 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:44.908317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:44.908347 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:44.908412 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:44.908556 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.909117 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.909946 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:44.910159 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:44.910220 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:44.910379 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:44.910525 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:44.910612 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:44.910630 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 02:53:44.910660 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:53:44.913390 14399 raft_consensus.cc:1261] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 1 FOLLOWER]: Refusing update from remote peer f203f484a2184864830664ae27717885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:44.913520 14661 raft_consensus.cc:1261] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 1 FOLLOWER]: Refusing update from remote peer f203f484a2184864830664ae27717885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:44.913925 14731 consensus_queue.cc:1035] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f5840cb97d49178bcc2e57718bd0fe" member_type: VOTER last_known_addr { host: "127.26.98.1" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:44.914137 14728 consensus_queue.cc:1035] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c96146cc4224c63901e8ee430a41907" member_type: VOTER last_known_addr { host: "127.26.98.3" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:44.917757 14736 mvcc.cc:205] Tried to move back new op lower bound from 7298406911641661440 to 7298406911550066688. Current Snapshot: MvccSnapshot[applied={T|T < 7298406911641661440}]
I20260619 02:53:44.918853 14735 mvcc.cc:205] Tried to move back new op lower bound from 7298406911641661440 to 7298406911550066688. Current Snapshot: MvccSnapshot[applied={T|T < 7298406911641661440}]
I20260619 02:53:44.920528 14257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:44.920691 14257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:44.921955 14257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf506851ec48410a90929213fb4d36d3: Sending DeleteTablet for 3 replicas of tablet 0d75808d58db43be9c0e12fe58a8f4ab
I20260619 02:53:44.922593 14379 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:44 UTC) from {username='slave'} at 127.0.0.1:41192
I20260619 02:53:44.922590 14641 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:44 UTC) from {username='slave'} at 127.0.0.1:42984
I20260619 02:53:44.923040 14744 tablet_replica.cc:330] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: stopping tablet replica
I20260619 02:53:44.923089 14745 tablet_replica.cc:330] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: stopping tablet replica
I20260619 02:53:44.923257 14744 raft_consensus.cc:2229] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:44.923280 14745 raft_consensus.cc:2229] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:44.923448 14744 raft_consensus.cc:2258] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:44.923390 14510 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d75808d58db43be9c0e12fe58a8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:44 UTC) from {username='slave'} at 127.0.0.1:55220
I20260619 02:53:44.923448 14745 raft_consensus.cc:2258] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:44.923691 14746 tablet_replica.cc:330] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: stopping tablet replica
I20260619 02:53:44.923826 14746 raft_consensus.cc:2229] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:44.923969 14746 raft_consensus.cc:2258] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:44.924536 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14317
I20260619 02:53:44.924621 14745 ts_tablet_manager.cc:1905] T 0d75808d58db43be9c0e12fe58a8f4ab P 35f5840cb97d49178bcc2e57718bd0fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:44.925263 14746 ts_tablet_manager.cc:1905] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:44.927276 14746 ts_tablet_manager.cc:1918] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:44.927490 14746 log.cc:1196] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-1/wal/wals/0d75808d58db43be9c0e12fe58a8f4ab
I20260619 02:53:44.927907 14746 ts_tablet_manager.cc:1939] T 0d75808d58db43be9c0e12fe58a8f4ab P f203f484a2184864830664ae27717885: Deleting consensus metadata
W20260619 02:53:44.931644 14243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.98.1:46475 (error 108)
I20260619 02:53:44.931725 14244 catalog_manager.cc:4841] TS f203f484a2184864830664ae27717885 (127.26.98.2:40535): tablet 0d75808d58db43be9c0e12fe58a8f4ab (table test-table [id=3d0d13e58bd94049b807d537d033ea5c]) successfully deleted
I20260619 02:53:44.932106 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14448
I20260619 02:53:44.932323 14744 ts_tablet_manager.cc:1905] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:44.932967 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:44.932224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.1:46475 (local address 127.0.0.1:41206)
0619 02:53:44.932862 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:44.932923 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.1:46475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
W20260619 02:53:44.933147 14243 catalog_manager.cc:4584] TS 35f5840cb97d49178bcc2e57718bd0fe (127.26.98.1:46475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d75808d58db43be9c0e12fe58a8f4ab: Network error: Client connection negotiation failed: client connection to 127.26.98.1:46475: connect: Connection refused (error 111)
I20260619 02:53:44.934650 14744 ts_tablet_manager.cc:1918] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:44.934756 14744 log.cc:1196] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Deleting WAL directory at /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837352638110-27016-0/minicluster-data/ts-2/wal/wals/0d75808d58db43be9c0e12fe58a8f4ab
I20260619 02:53:44.935032 14744 ts_tablet_manager.cc:1939] T 0d75808d58db43be9c0e12fe58a8f4ab P 5c96146cc4224c63901e8ee430a41907: Deleting consensus metadata
I20260619 02:53:44.935683 14242 catalog_manager.cc:4841] TS 5c96146cc4224c63901e8ee430a41907 (127.26.98.3:37267): tablet 0d75808d58db43be9c0e12fe58a8f4ab (table test-table [id=3d0d13e58bd94049b807d537d033ea5c]) successfully deleted
I20260619 02:53:44.939141 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14579
I20260619 02:53:44.944028 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14226
2026-06-19T02:53:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (974 ms)
[----------] 6 tests from EncryptionPolicyTest (9299 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:45 dist-test-slave-0f9d krb5kdc[14750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:45 dist-test-slave-0f9d krb5kdc[14750](info): set up 2 sockets
Jun 19 02:53:45 dist-test-slave-0f9d krb5kdc[14750](info): commencing operation
krb5kdc: starting...
W20260619 02:53:45.939193 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:46.073936 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:46.173382 14763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:46.173704 14763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:46.173745 14763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:46.175331 14763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:46.175509 14763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:46.175570 14763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:46.175597 14763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:46.175617 14763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:46.177896 14763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:46.178318 14763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:46.178642 14763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:46.184849 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:46.184854 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:46.184849 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:46.184885 14763 server_base.cc:1034] running on GCE node
I20260619 02:53:46.185444 14763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:46.185797 14763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:46.186975 14763 hybrid_clock.cc:648] HybridClock initialized: now 1781837626186948 us; error 45 us; skew 500 ppm
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:46.195184 14763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:46.195770 14763 webserver.cc:460] Webserver started at http://127.0.0.1:41899/ using document root <none> and password file <none>
I20260619 02:53:46.196035 14763 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:46.196089 14763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:46.196229 14763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:46.197518 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "e7a057f039fb4c2da8d8bc801496123c"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-0f9d"
server_key: "a5920ae5f779418ac72c2fe051390906"
server_key_iv: "31280928df9ae6d4dec1ad3dc3828901"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.197893 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "e7a057f039fb4c2da8d8bc801496123c"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-0f9d"
server_key: "a5920ae5f779418ac72c2fe051390906"
server_key_iv: "31280928df9ae6d4dec1ad3dc3828901"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.199551 14763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:46.200484 14778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:46.200722 14763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:46.200824 14763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "e7a057f039fb4c2da8d8bc801496123c"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-0f9d"
server_key: "a5920ae5f779418ac72c2fe051390906"
server_key_iv: "31280928df9ae6d4dec1ad3dc3828901"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.200901 14763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:46.226346 14763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:46.229581 14763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:46.229786 14763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:46.236222 14763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32803
I20260619 02:53:46.236276 14830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32803 every 8 connection(s)
I20260619 02:53:46.236722 14763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:46.237861 14831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:46.240911 14831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: Bootstrap starting.
I20260619 02:53:46.241137 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14763
I20260619 02:53:46.241284 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:46.241479 27016 external_mini_cluster.cc:1428] Setting key 8fb820cfdd536ba0ed0605ca7b13232c
I20260619 02:53:46.241801 14831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:46.242195 14831 log.cc:824] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:46.243280 14831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: No bootstrap required, opened a new log
I20260619 02:53:46.246086 14831 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } }
I20260619 02:53:46.246337 14831 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:46.246380 14831 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a057f039fb4c2da8d8bc801496123c, State: Initialized, Role: FOLLOWER
I20260619 02:53:46.246557 14831 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } }
I20260619 02:53:46.246654 14831 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:46.246699 14831 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:46.246757 14831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:46.247792 14831 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } }
I20260619 02:53:46.247953 14831 leader_election.cc:302] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a057f039fb4c2da8d8bc801496123c; no voters:
I20260619 02:53:46.248185 14831 leader_election.cc:288] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:46.248265 14836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:46.248451 14836 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [term 1 LEADER]: Becoming Leader. State: Replica: e7a057f039fb4c2da8d8bc801496123c, State: Running, Role: LEADER
I20260619 02:53:46.248594 14836 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } }
I20260619 02:53:46.248603 14831 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:46.249414 14837 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a057f039fb4c2da8d8bc801496123c. Latest consensus state: current_term: 1 leader_uuid: "e7a057f039fb4c2da8d8bc801496123c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } } }
I20260619 02:53:46.249660 14837 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:46.250711 14838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a057f039fb4c2da8d8bc801496123c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a057f039fb4c2da8d8bc801496123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32803 } } }
I20260619 02:53:46.250968 14838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:46.251127 14851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:46.251170 14851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:46.251231 14852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:46.251534 14852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:46.254293 14852 catalog_manager.cc:1347] Generated new cluster ID: dd93fb78a20e4d8ba56d2c6436c3aa8f
I20260619 02:53:46.254390 14852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:46.256695 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:46.242190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52279 (local address 127.0.0.1:32803)
0619 02:53:46.242442 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:46.242454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:46.242499 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:53:46.243344 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:46.243356 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:46.243446 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:46.243630 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:46.244347 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:46.246030 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:46.247236 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:46.247969 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:46.248129 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:46.252371 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:46.252378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:46.252379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:46.252394 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:46.254743 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:46.255310 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:46.255314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:46.255314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:46.255423 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:46.255665 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:46.255666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:46.255667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:46.255966 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:46.256155 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:46.256301 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:53:46.256425 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260619 02:53:46.258291 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:46.271454 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:46.260132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34817 (local address 127.0.0.1:32803)
0619 02:53:46.260184 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:46.260187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:46.263300 (+ 3113us) server_negotiation.cc:407] Connection header received
0619 02:53:46.263444 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:46.263446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:46.263462 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:46.263515 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:46.264234 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:46.264898 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:46.266035 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:46.266390 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:46.266492 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:46.268537 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:46.268540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:46.268540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:46.268552 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:46.270084 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:46.270552 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:46.270553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:46.270554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:46.270617 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:46.270823 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:46.270823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:46.270824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:46.271011 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:46.271123 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:46.271266 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:53:46.271337 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":24}
I20260619 02:53:46.279412 14852 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:46.280300 14852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:46.284961 14852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a057f039fb4c2da8d8bc801496123c: Generated new TSK 0
I20260619 02:53:46.285337 14852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:46.295634 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14763
W20260619 02:53:46.302229 14748 connection.cc:535] client connection to 127.0.0.1:32803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32803): Connection reset by peer (error 104)
2026-06-19T02:53:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14862](info): set up 2 sockets
Jun 19 02:53:46 dist-test-slave-0f9d krb5kdc[14862](info): commencing operation
krb5kdc: starting...
W20260619 02:53:47.299237 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:47 dist-test-slave-0f9d krb5kdc[14862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:47.443516 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:47.543244 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:47.543531 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:47.543560 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:47.544970 14875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:47.545051 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:47.545065 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:47.545077 14875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:47.545089 14875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:47.547278 14875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:47.547672 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:47.547950 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:47.553941 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:47.554116 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:47.554593 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:47.554666 14875 server_base.cc:1034] running on GCE node
I20260619 02:53:47.555126 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:47.555824 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:47.556999 14875 hybrid_clock.cc:648] HybridClock initialized: now 1781837627556978 us; error 44 us; skew 500 ppm
Jun 19 02:53:47 dist-test-slave-0f9d krb5kdc[14862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:47.561678 14875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:47.562515 14875 webserver.cc:460] Webserver started at http://127.0.0.1:35013/ using document root <none> and password file <none>
I20260619 02:53:47.562803 14875 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:47.562872 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:47.563102 14875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:47.564977 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "855b1e5211ac441f97ae75d0814d7bfa"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-0f9d"
server_key: "189ed48466b564660e938865e8419c67"
server_key_iv: "bd27d851bc8769c427cfbb2c99e8f06f"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.565575 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "855b1e5211ac441f97ae75d0814d7bfa"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-0f9d"
server_key: "189ed48466b564660e938865e8419c67"
server_key_iv: "bd27d851bc8769c427cfbb2c99e8f06f"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.567955 14875 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260619 02:53:47.569317 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:47.569603 14875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:47.569752 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "855b1e5211ac441f97ae75d0814d7bfa"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-0f9d"
server_key: "189ed48466b564660e938865e8419c67"
server_key_iv: "bd27d851bc8769c427cfbb2c99e8f06f"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.569873 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:47.590191 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:47.593762 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:47.593961 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:47.599989 14875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44043
I20260619 02:53:47.600076 14942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44043 every 8 connection(s)
I20260619 02:53:47.600560 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:47.601979 14943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:47.606722 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: Bootstrap starting.
I20260619 02:53:47.607915 14943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:47.608385 14943 log.cc:824] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:47.609472 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: No bootstrap required, opened a new log
I20260619 02:53:47.610699 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14875
I20260619 02:53:47.610859 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:47.611104 27016 external_mini_cluster.cc:1428] Setting key 32b4feae4c9f4e4c24b9a24fc26bb64d
I20260619 02:53:47.611750 14943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } }
I20260619 02:53:47.611969 14943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:47.612010 14943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855b1e5211ac441f97ae75d0814d7bfa, State: Initialized, Role: FOLLOWER
I20260619 02:53:47.612201 14943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } }
I20260619 02:53:47.612285 14943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:47.612319 14943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:47.612373 14943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:47.613147 14943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } }
I20260619 02:53:47.613274 14943 leader_election.cc:302] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 855b1e5211ac441f97ae75d0814d7bfa; no voters:
I20260619 02:53:47.613471 14943 leader_election.cc:288] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:47.613523 14948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:47.613662 14948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [term 1 LEADER]: Becoming Leader. State: Replica: 855b1e5211ac441f97ae75d0814d7bfa, State: Running, Role: LEADER
I20260619 02:53:47.613757 14948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } }
I20260619 02:53:47.613807 14943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:47.614176 14949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "855b1e5211ac441f97ae75d0814d7bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } } }
I20260619 02:53:47.614292 14949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:47.614169 14948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855b1e5211ac441f97ae75d0814d7bfa. Latest consensus state: current_term: 1 leader_uuid: "855b1e5211ac441f97ae75d0814d7bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855b1e5211ac441f97ae75d0814d7bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44043 } } }
I20260619 02:53:47.614493 14948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:47.615880 14963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:47.615957 14963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:47.617537 14955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:47.617817 14955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:47.618446 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:47.611797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57677 (local address 127.0.0.1:44043)
0619 02:53:47.612030 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:47.612042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:47.612125 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:53:47.612321 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:47.612405 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:47.612531 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:47.612742 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:47.613171 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.614510 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.615739 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.616325 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.616446 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:47.617500 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:47.617505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:47.617506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:47.617764 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:47.617817 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:47.617837 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:47.617931 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:47.618155 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:53:47.618189 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260619 02:53:47.619681 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:47.619998 14955 catalog_manager.cc:1347] Generated new cluster ID: 26847bf5192a4257845e49d4c77a0c10
I20260619 02:53:47.620047 14955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:47.626965 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:47.621488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50317 (local address 127.0.0.1:44043)
0619 02:53:47.621531 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:47.621534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:47.623107 (+ 1573us) server_negotiation.cc:407] Connection header received
0619 02:53:47.623184 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:47.623185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:47.623198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:47.623258 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:47.623433 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.624292 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.625385 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.625720 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.625784 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:47.626642 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:47.626643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:47.626643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:47.626650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:47.626669 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:47.626674 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:47.626726 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:47.626800 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:53:47.626818 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:47.630574 14955 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:47.631258 14955 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:47.633937 14955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 855b1e5211ac441f97ae75d0814d7bfa: Generated new TSK 0
I20260619 02:53:47.634166 14955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:47.639938 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14875
W20260619 02:53:47.644470 14860 connection.cc:535] client connection to 127.0.0.1:44043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44043): Connection reset by peer (error 104)
2026-06-19T02:53:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:47 dist-test-slave-0f9d krb5kdc[14974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:47 dist-test-slave-0f9d krb5kdc[14974](info): set up 2 sockets
Jun 19 02:53:47 dist-test-slave-0f9d krb5kdc[14974](info): commencing operation
krb5kdc: starting...
W20260619 02:53:48.639312 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:48 dist-test-slave-0f9d krb5kdc[14974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:48.782490 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:48.883436 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:48.883749 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:48.883782 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:48.885360 14987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:48.885419 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:48.885434 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:48.885445 14987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:48.885458 14987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:48.887352 14987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:48.887665 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:48.888034 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:48.895340 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:48.895340 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:48.895394 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:48.896041 14987 server_base.cc:1034] running on GCE node
I20260619 02:53:48.896579 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:48.896992 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:48.898191 14987 hybrid_clock.cc:648] HybridClock initialized: now 1781837628898172 us; error 43 us; skew 500 ppm
Jun 19 02:53:48 dist-test-slave-0f9d krb5kdc[14974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:48.901669 14987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:48.902298 14987 webserver.cc:460] Webserver started at http://127.0.0.1:46131/ using document root <none> and password file <none>
I20260619 02:53:48.902572 14987 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:48.902629 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:48.902770 14987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:48.904225 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "20b6b671bebc4bb3ae76db0754c04dee"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-0f9d"
server_key: "014e27800abe9434ce6b419ed7ab2520"
server_key_iv: "7bfc04308ab7ef257e87165ecacbe2d3"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.904652 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "20b6b671bebc4bb3ae76db0754c04dee"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-0f9d"
server_key: "014e27800abe9434ce6b419ed7ab2520"
server_key_iv: "7bfc04308ab7ef257e87165ecacbe2d3"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.906266 14987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:48.907770 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:48.908020 14987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:48.908135 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "20b6b671bebc4bb3ae76db0754c04dee"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-0f9d"
server_key: "014e27800abe9434ce6b419ed7ab2520"
server_key_iv: "7bfc04308ab7ef257e87165ecacbe2d3"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.908241 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:48.913120 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:48.915660 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:48.915795 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:48.920557 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43679
I20260619 02:53:48.920601 15054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43679 every 8 connection(s)
I20260619 02:53:48.920979 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:48.922008 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:48.925185 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee: Bootstrap starting.
I20260619 02:53:48.925998 15055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:48.926335 15055 log.cc:824] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:48.927431 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee: No bootstrap required, opened a new log
I20260619 02:53:48.929538 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 14987
I20260619 02:53:48.929657 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:48.929522 15055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 02:53:48.929751 15055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:48.929802 15055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b6b671bebc4bb3ae76db0754c04dee, State: Initialized, Role: FOLLOWER
I20260619 02:53:48.929874 27016 external_mini_cluster.cc:1428] Setting key 2b640daa2094be1ee4416bb4fd810f0a
I20260619 02:53:48.929991 15055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 02:53:48.930071 15055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:48.930124 15055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:48.930177 15055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:48.932003 15055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 02:53:48.932200 15055 leader_election.cc:302] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20b6b671bebc4bb3ae76db0754c04dee; no voters:
I20260619 02:53:48.932420 15055 leader_election.cc:288] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:48.932535 15060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:48.932713 15060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [term 1 LEADER]: Becoming Leader. State: Replica: 20b6b671bebc4bb3ae76db0754c04dee, State: Running, Role: LEADER
I20260619 02:53:48.932777 15055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:48.932796 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:48.930473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38499 (local address 127.0.0.1:43679)
0619 02:53:48.930701 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:48.930713 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:48.930748 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:53:48.931082 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:48.931091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:48.931136 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:48.931290 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:48.932127 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:48.932132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:48.932132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:48.932136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:48.932217 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:48.932237 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:48.932319 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:48.932450 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:53:48.932549 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260619 02:53:48.932934 15060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 02:53:48.933524 15062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20b6b671bebc4bb3ae76db0754c04dee. Latest consensus state: current_term: 1 leader_uuid: "20b6b671bebc4bb3ae76db0754c04dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260619 02:53:48.933525 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20b6b671bebc4bb3ae76db0754c04dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b6b671bebc4bb3ae76db0754c04dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260619 02:53:48.933676 15062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:48.933677 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:48.933911 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:48.934734 15067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:48.935393 15067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:48.937549 15067 catalog_manager.cc:1347] Generated new cluster ID: 52378b1361944aa68925f3231c2bae56
I20260619 02:53:48.937629 15067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:48.944898 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:48.939380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53995 (local address 127.0.0.1:43679)
0619 02:53:48.939456 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:48.939459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:48.943219 (+ 3760us) server_negotiation.cc:407] Connection header received
0619 02:53:48.943341 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:48.943342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:48.943369 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:48.943426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:48.944574 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:48.944575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:48.944576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:48.944581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:48.944606 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:48.944612 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:48.944695 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:48.944772 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:53:48.944791 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:53:48.953377 15067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:48.953634 15067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:48.966784 15067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20b6b671bebc4bb3ae76db0754c04dee: Generated new TSK 0
I20260619 02:53:48.967124 15067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:48.970839 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 14987
2026-06-19T02:53:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:49 dist-test-slave-0f9d krb5kdc[15086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:49 dist-test-slave-0f9d krb5kdc[15086](info): set up 2 sockets
Jun 19 02:53:49 dist-test-slave-0f9d krb5kdc[15086](info): commencing operation
krb5kdc: starting...
W20260619 02:53:49.975193 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:50.124482 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:50.229354 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:50.229733 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:50.229777 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:50.231516 15099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:50.231601 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:50.231623 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:50.231642 15099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:50.231667 15099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:50.234076 15099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:50.234488 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:50.234802 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:50.241109 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:50.241097 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:50.241086 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:50.241636 15099 server_base.cc:1034] running on GCE node
I20260619 02:53:50.241925 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:50.242293 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:50.243498 15099 hybrid_clock.cc:648] HybridClock initialized: now 1781837630243467 us; error 56 us; skew 500 ppm
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:50.247514 15099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:50.248087 15099 webserver.cc:460] Webserver started at http://127.0.0.1:39407/ using document root <none> and password file <none>
I20260619 02:53:50.248374 15099 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:50.248431 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:50.248569 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:50.249737 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "0f37443596b64f90971af440b2038385"
format_stamp: "Formatted at 2026-06-19 02:53:50 on dist-test-slave-0f9d"
server_key: "c218f32ada0fc5beb0619de12909e3b4"
server_key_iv: "0744f39af87496591c72b8dd0f028cda"
server_key_version: "encryptionkey@0"
I20260619 02:53:50.250083 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "0f37443596b64f90971af440b2038385"
format_stamp: "Formatted at 2026-06-19 02:53:50 on dist-test-slave-0f9d"
server_key: "c218f32ada0fc5beb0619de12909e3b4"
server_key_iv: "0744f39af87496591c72b8dd0f028cda"
server_key_version: "encryptionkey@0"
I20260619 02:53:50.252471 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:53:50.253635 15114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:50.253897 15099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:50.254015 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0f37443596b64f90971af440b2038385"
format_stamp: "Formatted at 2026-06-19 02:53:50 on dist-test-slave-0f9d"
server_key: "c218f32ada0fc5beb0619de12909e3b4"
server_key_iv: "0744f39af87496591c72b8dd0f028cda"
server_key_version: "encryptionkey@0"
I20260619 02:53:50.254107 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:50.279560 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:50.282801 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:50.283032 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:50.287864 15099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34611
I20260619 02:53:50.287912 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34611 every 8 connection(s)
I20260619 02:53:50.288326 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:50.289420 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:50.292419 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385: Bootstrap starting.
I20260619 02:53:50.292469 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15099
I20260619 02:53:50.292598 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:50.292817 27016 external_mini_cluster.cc:1428] Setting key e832d900f025ef949a4bb7cb0323c99e
I20260619 02:53:50.293227 15167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:50.293642 15167 log.cc:824] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:50.294456 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385: No bootstrap required, opened a new log
I20260619 02:53:50.296983 15167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260619 02:53:50.297248 15167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:50.297289 15167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f37443596b64f90971af440b2038385, State: Initialized, Role: FOLLOWER
I20260619 02:53:50.297456 15167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260619 02:53:50.297540 15167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:50.297572 15167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:50.297634 15167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:50.298527 15167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260619 02:53:50.298800 15167 leader_election.cc:302] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f37443596b64f90971af440b2038385; no voters:
I20260619 02:53:50.299064 15167 leader_election.cc:288] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:50.299136 15172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:50.299350 15172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [term 1 LEADER]: Becoming Leader. State: Replica: 0f37443596b64f90971af440b2038385, State: Running, Role: LEADER
I20260619 02:53:50.299407 15167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:50.299554 15172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260619 02:53:50.300026 15174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f37443596b64f90971af440b2038385. Latest consensus state: current_term: 1 leader_uuid: "0f37443596b64f90971af440b2038385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260619 02:53:50.300127 15174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:50.300830 15180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:50.300933 15173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f37443596b64f90971af440b2038385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f37443596b64f90971af440b2038385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260619 02:53:50.301028 15173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385 [sys.catalog]: This master's current role is: LEADER
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:50.301771 15180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:50.304611 15180 catalog_manager.cc:1347] Generated new cluster ID: 15c0afc9462b4746a6381c355cf33870
I20260619 02:53:50.304697 15180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:50.307418 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:50.293476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59813 (local address 127.0.0.1:34611)
0619 02:53:50.293682 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:50.293693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:50.293729 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:53:50.294378 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:50.294386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:50.294447 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:50.294615 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:50.295680 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:50.296716 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:50.297822 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:50.298342 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:50.298453 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:50.302051 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:50.302056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:50.302057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:50.302073 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:50.304679 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:50.305219 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:50.305223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:50.305223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:50.305340 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:50.305536 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:50.305537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:50.305537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:50.305896 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:50.306086 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:50.306791 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 02:53:50.307119 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260619 02:53:50.316601 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:50.327709 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:50.318505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45391 (local address 127.0.0.1:34611)
0619 02:53:50.318563 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:50.318566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:50.318739 (+ 173us) server_negotiation.cc:407] Connection header received
0619 02:53:50.318809 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:50.318810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:50.318823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:50.318918 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:50.319773 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:50.320594 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:50.321821 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:50.322166 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:50.322235 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:50.324060 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:50.324063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:50.324063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:50.324072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:50.325998 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:50.326544 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:50.326545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:50.326546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:50.326634 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:50.327001 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:50.327001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:50.327002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:50.327155 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:50.327270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:50.327463 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:53:50.327584 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:53:50.328979 15180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:50.329603 15180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:50.334085 15180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f37443596b64f90971af440b2038385: Generated new TSK 0
I20260619 02:53:50.334470 15180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:50.340255 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15099
2026-06-19T02:53:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15198](info): set up 2 sockets
Jun 19 02:53:50 dist-test-slave-0f9d krb5kdc[15198](info): commencing operation
krb5kdc: starting...
W20260619 02:53:51.351861 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:51 dist-test-slave-0f9d krb5kdc[15198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:51.499464 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:51.600271 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:51.600590 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:51.600636 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:51.602254 15211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:51.602336 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:51.602355 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:51.602375 15211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:51.602389 15211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:51.604558 15211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:51.604941 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:51.605226 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:51.611979 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:51.612133 15211 server_base.cc:1034] running on GCE node
W20260619 02:53:51.611984 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:51.611979 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:51.612620 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:51.612999 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:51.614308 15211 hybrid_clock.cc:648] HybridClock initialized: now 1781837631614213 us; error 44 us; skew 500 ppm
Jun 19 02:53:51 dist-test-slave-0f9d krb5kdc[15198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:51.622629 15211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:51.623445 15211 webserver.cc:460] Webserver started at http://127.0.0.1:34273/ using document root <none> and password file <none>
I20260619 02:53:51.623745 15211 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:51.623808 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:51.623965 15211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:51.625339 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "5d167a992b764198ab0cd48f08acade4"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-0f9d"
server_key: "cf9d95af0c57805a05236bcc22ff66f1"
server_key_iv: "4e8a63d05e1cc5ec44e1e27e2cc386cf"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.625758 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "5d167a992b764198ab0cd48f08acade4"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-0f9d"
server_key: "cf9d95af0c57805a05236bcc22ff66f1"
server_key_iv: "4e8a63d05e1cc5ec44e1e27e2cc386cf"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.627593 15211 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 02:53:51.628758 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:51.629172 15211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:51.629319 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "5d167a992b764198ab0cd48f08acade4"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-0f9d"
server_key: "cf9d95af0c57805a05236bcc22ff66f1"
server_key_iv: "4e8a63d05e1cc5ec44e1e27e2cc386cf"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.629420 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:51.647332 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:51.650378 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:51.650569 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:51.656036 15211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41565
I20260619 02:53:51.656098 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41565 every 8 connection(s)
I20260619 02:53:51.656978 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:51.657214 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15211
I20260619 02:53:51.657363 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:51.657603 27016 external_mini_cluster.cc:1428] Setting key e5b7bf85267daa702f0941e608d54cdb
I20260619 02:53:51.658326 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:51.662775 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: Bootstrap starting.
I20260619 02:53:51.663774 15279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:51.664324 15279 log.cc:824] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:51.665349 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: No bootstrap required, opened a new log
I20260619 02:53:51.665395 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:51.658335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41539 (local address 127.0.0.1:41565)
0619 02:53:51.658589 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:51.658601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:51.658638 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:53:51.659401 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:51.659416 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:51.659509 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:51.659749 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:51.660092 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.661786 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.663069 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.663650 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.663792 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:51.664626 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:51.664632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:51.664633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:51.664847 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:51.664891 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:51.664907 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:51.664975 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:51.665056 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:53:51.665149 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":145,"threads_started":1}
I20260619 02:53:51.666642 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:51.667989 15279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260619 02:53:51.668222 15279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:51.668262 15279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d167a992b764198ab0cd48f08acade4, State: Initialized, Role: FOLLOWER
I20260619 02:53:51.668426 15279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260619 02:53:51.668509 15279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:51.668561 15279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:51.668619 15279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:51.669406 15279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260619 02:53:51.669554 15279 leader_election.cc:302] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d167a992b764198ab0cd48f08acade4; no voters:
I20260619 02:53:51.669754 15279 leader_election.cc:288] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:51.669821 15289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:51.669998 15289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d167a992b764198ab0cd48f08acade4, State: Running, Role: LEADER
I20260619 02:53:51.670094 15279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:51.670128 15289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260619 02:53:51.671241 15291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d167a992b764198ab0cd48f08acade4. Latest consensus state: current_term: 1 leader_uuid: "5d167a992b764198ab0cd48f08acade4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260619 02:53:51.671700 15290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d167a992b764198ab0cd48f08acade4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d167a992b764198ab0cd48f08acade4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260619 02:53:51.672149 15290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:51.671710 15291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4 [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:51.672369 15304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:51.672442 15304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:51.672557 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:51.668287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53911 (local address 127.0.0.1:41565)
0619 02:53:51.668344 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:51.668347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:51.668484 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:53:51.668555 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:51.668556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:51.668568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:51.668615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:51.668760 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.669312 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.670368 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.670721 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.670817 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:51.672001 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:51.672003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:51.672004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:51.672012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:51.672034 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:51.672040 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:51.672098 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:51.672433 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 02:53:51.672476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:53:51.675155 15306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:51.675487 15306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:51.677691 15306 catalog_manager.cc:1347] Generated new cluster ID: 89f946c07be548cc8775522ba9d7a9df
I20260619 02:53:51.677803 15306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:51.689540 15306 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:51.690948 15306 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:51.696239 15306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d167a992b764198ab0cd48f08acade4: Generated new TSK 0
I20260619 02:53:51.696589 15306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:51.700582 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15211
2026-06-19T02:53:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:51 dist-test-slave-0f9d krb5kdc[15310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:51 dist-test-slave-0f9d krb5kdc[15310](info): set up 2 sockets
Jun 19 02:53:51 dist-test-slave-0f9d krb5kdc[15310](info): commencing operation
krb5kdc: starting...
W20260619 02:53:52.699611 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:52 dist-test-slave-0f9d krb5kdc[15310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:52.834312 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:52.927983 15323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:52.928246 15323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:52.928284 15323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:52.929587 15323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:52.929644 15323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:52.929663 15323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:52.929680 15323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:52.929695 15323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:52.931404 15323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:52.931728 15323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:52.931988 15323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:52.937822 15329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:52.937848 15331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:52.937812 15328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:52.938026 15323 server_base.cc:1034] running on GCE node
I20260619 02:53:52.938347 15323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:52.938711 15323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:52.939881 15323 hybrid_clock.cc:648] HybridClock initialized: now 1781837632939856 us; error 41 us; skew 500 ppm
Jun 19 02:53:52 dist-test-slave-0f9d krb5kdc[15310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:52.943322 15323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:52.943957 15323 webserver.cc:460] Webserver started at http://127.0.0.1:33909/ using document root <none> and password file <none>
I20260619 02:53:52.944243 15323 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:52.944295 15323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:52.944438 15323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:52.945771 15323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "ba4b03c08fc14a3388decaeb69d8fd54"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-0f9d"
server_key: "2a8d1e8995182f4ba410f8b71043c14c"
server_key_iv: "60bd88e99844aaff51bc63b7a6919d37"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.946156 15323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "ba4b03c08fc14a3388decaeb69d8fd54"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-0f9d"
server_key: "2a8d1e8995182f4ba410f8b71043c14c"
server_key_iv: "60bd88e99844aaff51bc63b7a6919d37"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.947846 15323 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:53:52.948947 15338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:52.949311 15323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:52.949442 15323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "ba4b03c08fc14a3388decaeb69d8fd54"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-0f9d"
server_key: "2a8d1e8995182f4ba410f8b71043c14c"
server_key_iv: "60bd88e99844aaff51bc63b7a6919d37"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.949551 15323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:52.954653 15323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:52.957710 15323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:52.957896 15323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:52.962771 15323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42633
I20260619 02:53:52.962852 15390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42633 every 8 connection(s)
I20260619 02:53:52.963275 15323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:52.964586 15391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:52.967648 15391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54: Bootstrap starting.
I20260619 02:53:52.968371 15391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:52.968658 15391 log.cc:824] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:52.969635 15391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54: No bootstrap required, opened a new log
I20260619 02:53:52.970695 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15323
I20260619 02:53:52.970815 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:52.971033 27016 external_mini_cluster.cc:1428] Setting key 00a734a3bf3205618e3ad29d3a69eb66
I20260619 02:53:52.971957 15391 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260619 02:53:52.972188 15391 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:52.972224 15391 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4b03c08fc14a3388decaeb69d8fd54, State: Initialized, Role: FOLLOWER
I20260619 02:53:52.972383 15391 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260619 02:53:52.972481 15391 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:52.972522 15391 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:52.972575 15391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:52.973524 15391 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260619 02:53:52.973646 15391 leader_election.cc:302] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4b03c08fc14a3388decaeb69d8fd54; no voters:
I20260619 02:53:52.973798 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:52.971698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34969 (local address 127.0.0.1:42633)
0619 02:53:52.971929 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:52.971941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:52.971978 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:53:52.972117 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:52.972126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:52.972180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:52.972664 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:52.973304 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:52.973308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:52.973309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:52.973313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:52.973358 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:52.973374 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:52.973460 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:52.973497 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 02:53:52.973544 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260619 02:53:52.973850 15391 leader_election.cc:288] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:52.973917 15396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:52.974061 15396 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [term 1 LEADER]: Becoming Leader. State: Replica: ba4b03c08fc14a3388decaeb69d8fd54, State: Running, Role: LEADER
I20260619 02:53:52.974169 15391 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:52.974171 15396 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260619 02:53:52.974807 15397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4b03c08fc14a3388decaeb69d8fd54. Latest consensus state: current_term: 1 leader_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } } }
I20260619 02:53:52.974931 15397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:52.974856 15398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b03c08fc14a3388decaeb69d8fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } } }
I20260619 02:53:52.975113 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:52.975131 15398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:52.976248 15405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:52.977133 15405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:52.979125 15405 catalog_manager.cc:1347] Generated new cluster ID: b7a30060885f43d3b4d6668e5da8d29e
I20260619 02:53:52.979195 15405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:52.984474 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:52.977239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40705 (local address 127.0.0.1:42633)
0619 02:53:52.977284 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:52.977286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:52.983132 (+ 5846us) server_negotiation.cc:407] Connection header received
0619 02:53:52.983264 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:52.983266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:52.983279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:52.983348 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:52.984202 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:52.984204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:52.984205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:52.984212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:52.984233 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:52.984238 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:52.984310 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:52.984352 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:53:52.984371 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:52.998008 15405 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:52.998270 15405 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:53.001183 15405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4b03c08fc14a3388decaeb69d8fd54: Generated new TSK 0
I20260619 02:53:53.001483 15405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:53.007819 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15323
2026-06-19T02:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:53 dist-test-slave-0f9d krb5kdc[15422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:53 dist-test-slave-0f9d krb5kdc[15422](info): set up 2 sockets
Jun 19 02:53:53 dist-test-slave-0f9d krb5kdc[15422](info): commencing operation
krb5kdc: starting...
W20260619 02:53:53.991503 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:54.140193 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:54.239524 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:54.239827 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:54.239858 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:54.241436 15435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:54.241510 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:54.241526 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:54.241539 15435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:54.241550 15435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:54.243570 15435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:54.243911 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:54.244174 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:54.250415 15435 server_base.cc:1034] running on GCE node
W20260619 02:53:54.250386 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:54.250578 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:54.250394 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:54.250849 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:54.251282 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:54.252497 15435 hybrid_clock.cc:648] HybridClock initialized: now 1781837634252469 us; error 38 us; skew 500 ppm
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:54.256108 15435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:54.256805 15435 webserver.cc:460] Webserver started at http://127.0.0.1:45305/ using document root <none> and password file <none>
I20260619 02:53:54.257056 15435 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:54.257105 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:54.257218 15435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:54.258497 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "0d39ae125a0748a3bb6cef00c3598f38"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-0f9d"
server_key: "de34ad616ceafcee62f9ebfea6513784"
server_key_iv: "ba585694cbdd6964b872f2c7ef85462f"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.258903 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "0d39ae125a0748a3bb6cef00c3598f38"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-0f9d"
server_key: "de34ad616ceafcee62f9ebfea6513784"
server_key_iv: "ba585694cbdd6964b872f2c7ef85462f"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.260646 15435 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:54.261911 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:54.262161 15435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:54.262261 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "0d39ae125a0748a3bb6cef00c3598f38"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-0f9d"
server_key: "de34ad616ceafcee62f9ebfea6513784"
server_key_iv: "ba585694cbdd6964b872f2c7ef85462f"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.262444 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:54.291209 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:54.294305 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:54.294476 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:54.300019 15435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39929
I20260619 02:53:54.300061 15502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39929 every 8 connection(s)
I20260619 02:53:54.300732 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:54.301867 15503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:54.305059 15503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38: Bootstrap starting.
I20260619 02:53:54.305850 15503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:54.306180 15503 log.cc:824] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:54.307263 15503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38: No bootstrap required, opened a new log
I20260619 02:53:54.308023 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15435
I20260619 02:53:54.308157 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:54.308334 27016 external_mini_cluster.cc:1428] Setting key f41e874b46c0d6c448d3c1d48c7b1dae
I20260619 02:53:54.310477 15503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260619 02:53:54.310745 15503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:54.310786 15503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d39ae125a0748a3bb6cef00c3598f38, State: Initialized, Role: FOLLOWER
I20260619 02:53:54.311215 15503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260619 02:53:54.311379 15503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:54.311422 15503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:54.311491 15503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:54.312673 15503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260619 02:53:54.312811 15503 leader_election.cc:302] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d39ae125a0748a3bb6cef00c3598f38; no voters:
I20260619 02:53:54.313153 15503 leader_election.cc:288] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:54.313189 15508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:54.313323 15508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [term 1 LEADER]: Becoming Leader. State: Replica: 0d39ae125a0748a3bb6cef00c3598f38, State: Running, Role: LEADER
I20260619 02:53:54.313421 15508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260619 02:53:54.313530 15503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:54.313853 15510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d39ae125a0748a3bb6cef00c3598f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20260619 02:53:54.313864 15509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d39ae125a0748a3bb6cef00c3598f38. Latest consensus state: current_term: 1 leader_uuid: "0d39ae125a0748a3bb6cef00c3598f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d39ae125a0748a3bb6cef00c3598f38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20260619 02:53:54.314213 15509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:54.314033 15510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:54.315028 15513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:54.315768 15513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:54.317839 15513 catalog_manager.cc:1347] Generated new cluster ID: c403f54b4ddd4a848e1d6048b3860bb0
I20260619 02:53:54.317898 15513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:54.323280 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.309103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:34095 (local address 127.0.0.1:39929)
0619 02:53:54.309446 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:54.309458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:54.309499 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:53:54.310299 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:54.310359 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:54.310453 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:54.310637 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:54.311584 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.313178 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.314504 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.315208 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.315324 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.318660 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:54.318666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:54.318666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:54.318681 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:54.321100 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.321734 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.321738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.321738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.321906 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.322172 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.322172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.322173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.322541 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:54.322727 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:54.322861 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:53:54.323025 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":194,"threads_started":1}
I20260619 02:53:54.324579 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:54.329011 15513 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:54.329725 15513 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:54.335000 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.326675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:51291 (local address 127.0.0.1:39929)
0619 02:53:54.326723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:54.326725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:54.326737 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:54.326765 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:54.326766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:54.326777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:54.326839 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:54.327599 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.328504 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.329522 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.329883 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.329955 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.331947 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:54.331950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:54.331950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:54.331960 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:54.333664 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.334113 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.334114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.334114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.334183 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.334382 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.334383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.334383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.334537 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:54.334666 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:54.334767 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 02:53:54.334841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:54.336766 15513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d39ae125a0748a3bb6cef00c3598f38: Generated new TSK 0
I20260619 02:53:54.337091 15513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:54.340698 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15435
W20260619 02:53:54.345526 15420 connection.cc:535] client connection to 127.0.0.1:39929 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39929): Connection reset by peer (error 104)
2026-06-19T02:53:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15534](info): set up 2 sockets
Jun 19 02:53:54 dist-test-slave-0f9d krb5kdc[15534](info): commencing operation
krb5kdc: starting...
W20260619 02:53:55.331481 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:55 dist-test-slave-0f9d krb5kdc[15534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:55.465603 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:55.566037 15547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:55.566370 15547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:55.566411 15547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:55.568118 15547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:55.568221 15547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:55.568246 15547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:55.568264 15547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:55.568279 15547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:55.570371 15547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:55.570749 15547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:55.571095 15547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:55.577323 15552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:55.577504 15547 server_base.cc:1034] running on GCE node
W20260619 02:53:55.577358 15555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:55.577358 15553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:55.577914 15547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:55.578274 15547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:55.579460 15547 hybrid_clock.cc:648] HybridClock initialized: now 1781837635579444 us; error 35 us; skew 500 ppm
Jun 19 02:53:55 dist-test-slave-0f9d krb5kdc[15534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:55.582496 15547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:55.583094 15547 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260619 02:53:55.583334 15547 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:55.583389 15547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:55.583547 15547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:55.584851 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "58626604e4584ec39acf380d720cfe7b"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-0f9d"
server_key: "77219e1aeb240559dea248237569d439"
server_key_iv: "70841fc369d1d64641dd1d8f73b5a9cb"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.585242 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "58626604e4584ec39acf380d720cfe7b"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-0f9d"
server_key: "77219e1aeb240559dea248237569d439"
server_key_iv: "70841fc369d1d64641dd1d8f73b5a9cb"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.586814 15547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:55.587809 15562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:55.588230 15547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:55.588361 15547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "58626604e4584ec39acf380d720cfe7b"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-0f9d"
server_key: "77219e1aeb240559dea248237569d439"
server_key_iv: "70841fc369d1d64641dd1d8f73b5a9cb"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.588451 15547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:55.614853 15547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:55.617504 15547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:55.617697 15547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:55.622496 15547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40647
I20260619 02:53:55.622536 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40647 every 8 connection(s)
I20260619 02:53:55.623015 15547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:55.624007 15615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:55.627081 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b: Bootstrap starting.
I20260619 02:53:55.627797 15615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:55.628067 15615 log.cc:824] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:55.628890 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b: No bootstrap required, opened a new log
I20260619 02:53:55.630672 15615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260619 02:53:55.630843 15615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:55.630867 15615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58626604e4584ec39acf380d720cfe7b, State: Initialized, Role: FOLLOWER
I20260619 02:53:55.631053 15615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260619 02:53:55.631124 15615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:55.631153 15615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:55.631189 15615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:55.631784 15615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260619 02:53:55.631878 15615 leader_election.cc:302] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58626604e4584ec39acf380d720cfe7b; no voters:
I20260619 02:53:55.632035 15615 leader_election.cc:288] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:55.632084 15618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:55.632258 15618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [term 1 LEADER]: Becoming Leader. State: Replica: 58626604e4584ec39acf380d720cfe7b, State: Running, Role: LEADER
I20260619 02:53:55.632337 15615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:55.632385 15618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260619 02:53:55.632697 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15547
I20260619 02:53:55.632822 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:55.633004 27016 external_mini_cluster.cc:1428] Setting key 5d0bb430c10e2f73f48862095f43fe13
I20260619 02:53:55.633023 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58626604e4584ec39acf380d720cfe7b. Latest consensus state: current_term: 1 leader_uuid: "58626604e4584ec39acf380d720cfe7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } } }
I20260619 02:53:55.633091 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:55.633239 15619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58626604e4584ec39acf380d720cfe7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58626604e4584ec39acf380d720cfe7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } } }
I20260619 02:53:55.633729 15627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:55.633960 15619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:55.634472 15627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:55.636098 15627 catalog_manager.cc:1347] Generated new cluster ID: b2629247649a43a19e88712a038b189a
I20260619 02:53:55.636155 15627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:55.640576 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.633603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:52997 (local address 127.0.0.1:40647)
0619 02:53:55.633974 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.633986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.634027 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:53:55.634167 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.634174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.634229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.634391 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.634622 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.636120 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.637454 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.638147 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.638263 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.639370 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.639376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.639376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:55.639644 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.639772 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:55.639808 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.639919 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.640233 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.640273 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":100,"threads_started":1}
I20260619 02:53:55.641848 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:55.642037 15627 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:55.642577 15627 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:55.645596 15627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58626604e4584ec39acf380d720cfe7b: Generated new TSK 0
I20260619 02:53:55.645839 15627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:55.647925 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.643872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:42585 (local address 127.0.0.1:40647)
0619 02:53:55.643929 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.643931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.644001 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:53:55.644072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.644072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.644085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.644160 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.644326 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.645208 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.646405 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.646636 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.646702 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.647606 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.647607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.647608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:55.647615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.647634 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:55.647639 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.647704 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.647791 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.647806 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:53:55.650966 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15547
W20260619 02:53:55.655802 15532 connection.cc:535] client connection to 127.0.0.1:40647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40647): Connection reset by peer (error 104)
2026-06-19T02:53:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:55 dist-test-slave-0f9d krb5kdc[15646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:55 dist-test-slave-0f9d krb5kdc[15646](info): set up 2 sockets
Jun 19 02:53:55 dist-test-slave-0f9d krb5kdc[15646](info): commencing operation
krb5kdc: starting...
W20260619 02:53:56.635304 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:56 dist-test-slave-0f9d krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:56.777577 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:56.880820 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:56.881148 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:56.881191 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:56.882784 15659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:56.882925 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:56.882952 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:56.882975 15659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:56.883000 15659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:56.885191 15659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:56.885589 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:56.885879 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:56.891690 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:56.891690 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:56.891690 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:56.892061 15659 server_base.cc:1034] running on GCE node
I20260619 02:53:56.892359 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:56.892714 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:56.893955 15659 hybrid_clock.cc:648] HybridClock initialized: now 1781837636893918 us; error 38 us; skew 500 ppm
Jun 19 02:53:56 dist-test-slave-0f9d krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:56.902783 15659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:56.903489 15659 webserver.cc:460] Webserver started at http://127.0.0.1:43455/ using document root <none> and password file <none>
I20260619 02:53:56.903796 15659 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:56.903862 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:56.904053 15659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:56.905440 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "737a4b5f680f46b2a07100093829a39a"
format_stamp: "Formatted at 2026-06-19 02:53:56 on dist-test-slave-0f9d"
server_key: "01da035aaca234f3b5c7b6460b3c872a"
server_key_iv: "0cf308ae4e3fffb3baf017e4ea90ba73"
server_key_version: "encryptionkey@0"
I20260619 02:53:56.905850 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "737a4b5f680f46b2a07100093829a39a"
format_stamp: "Formatted at 2026-06-19 02:53:56 on dist-test-slave-0f9d"
server_key: "01da035aaca234f3b5c7b6460b3c872a"
server_key_iv: "0cf308ae4e3fffb3baf017e4ea90ba73"
server_key_version: "encryptionkey@0"
I20260619 02:53:56.907478 15659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 02:53:56.908578 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:56.908821 15659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:56.908922 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "737a4b5f680f46b2a07100093829a39a"
format_stamp: "Formatted at 2026-06-19 02:53:56 on dist-test-slave-0f9d"
server_key: "01da035aaca234f3b5c7b6460b3c872a"
server_key_iv: "0cf308ae4e3fffb3baf017e4ea90ba73"
server_key_version: "encryptionkey@0"
I20260619 02:53:56.909003 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:56.913169 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:56.916198 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:56.916378 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:56.921324 15659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34929
I20260619 02:53:56.921403 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34929 every 8 connection(s)
I20260619 02:53:56.921958 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:56.923269 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:56.924842 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15659
I20260619 02:53:56.925007 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:56.925196 27016 external_mini_cluster.cc:1428] Setting key 2bf0297086881ed99fed9c6c2116ad00
I20260619 02:53:56.926930 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a: Bootstrap starting.
I20260619 02:53:56.927681 15727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:56.928041 15727 log.cc:824] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:56.928738 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.925954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:53051 (local address 127.0.0.1:34929)
0619 02:53:56.926263 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.926274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.926310 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:53:56.926990 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.926999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.927038 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.927344 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:56.928147 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:56.928152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:56.928152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:56.928158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:56.928217 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:56.928232 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:56.928315 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.928399 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.928496 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260619 02:53:56.928946 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a: No bootstrap required, opened a new log
I20260619 02:53:56.930248 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:56.931434 15727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } }
I20260619 02:53:56.931669 15727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:56.931710 15727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737a4b5f680f46b2a07100093829a39a, State: Initialized, Role: FOLLOWER
I20260619 02:53:56.931869 15727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } }
I20260619 02:53:56.931982 15727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:56.932053 15727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:56.932116 15727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:56.933144 15727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } }
I20260619 02:53:56.933257 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.931975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:43995 (local address 127.0.0.1:34929)
0619 02:53:56.932030 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.932032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.932193 (+ 161us) server_negotiation.cc:407] Connection header received
0619 02:53:56.932270 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.932271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.932279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.932328 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:56.933034 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:56.933036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:56.933036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:56.933047 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:56.933066 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:56.933072 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:56.933121 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.933168 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.933185 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:53:56.933357 15727 leader_election.cc:302] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737a4b5f680f46b2a07100093829a39a; no voters:
I20260619 02:53:56.933620 15737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:56.933617 15727 leader_election.cc:288] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:56.933811 15737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [term 1 LEADER]: Becoming Leader. State: Replica: 737a4b5f680f46b2a07100093829a39a, State: Running, Role: LEADER
I20260619 02:53:56.933974 15737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } }
I20260619 02:53:56.934016 15727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:56.934481 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737a4b5f680f46b2a07100093829a39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } } }
I20260619 02:53:56.934612 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:56.934455 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737a4b5f680f46b2a07100093829a39a. Latest consensus state: current_term: 1 leader_uuid: "737a4b5f680f46b2a07100093829a39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a4b5f680f46b2a07100093829a39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34929 } } }
I20260619 02:53:56.934857 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:56.935372 15746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:56.935962 15746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:56.939000 15746 catalog_manager.cc:1347] Generated new cluster ID: c773342de90c4f42b170ed54046205cb
I20260619 02:53:56.939131 15746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:56.947705 15746 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:56.948007 15746 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:56.953533 15746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737a4b5f680f46b2a07100093829a39a: Generated new TSK 0
I20260619 02:53:56.953797 15746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:56.964334 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15659
2026-06-19T02:53:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:57 dist-test-slave-0f9d krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:57 dist-test-slave-0f9d krb5kdc[15758](info): set up 2 sockets
Jun 19 02:53:57 dist-test-slave-0f9d krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260619 02:53:57.967624 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:58.107618 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:58.209085 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:58.209400 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:58.209429 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:58.210925 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:58.210996 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:58.211012 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:58.211024 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:58.211035 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:58.212942 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:58.213245 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:58.213542 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:58.219271 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:58.219271 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:58.219271 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:58.219560 15771 server_base.cc:1034] running on GCE node
I20260619 02:53:58.219847 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:58.220196 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:58.221385 15771 hybrid_clock.cc:648] HybridClock initialized: now 1781837638221363 us; error 31 us; skew 500 ppm
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:58.225208 15771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:58.225826 15771 webserver.cc:460] Webserver started at http://127.0.0.1:45423/ using document root <none> and password file <none>
I20260619 02:53:58.226102 15771 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:58.226162 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:58.226313 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:58.227677 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "7f946f086c084ca283b55091374cfc6f"
format_stamp: "Formatted at 2026-06-19 02:53:58 on dist-test-slave-0f9d"
server_key: "6a0915fa4951a2c6e4ee8b0200790ff0"
server_key_iv: "17bfa2d47662ffbab6ab101f509482cd"
server_key_version: "encryptionkey@0"
I20260619 02:53:58.228102 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "7f946f086c084ca283b55091374cfc6f"
format_stamp: "Formatted at 2026-06-19 02:53:58 on dist-test-slave-0f9d"
server_key: "6a0915fa4951a2c6e4ee8b0200790ff0"
server_key_iv: "17bfa2d47662ffbab6ab101f509482cd"
server_key_version: "encryptionkey@0"
I20260619 02:53:58.230082 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:58.231245 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:58.231623 15771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:58.231801 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "7f946f086c084ca283b55091374cfc6f"
format_stamp: "Formatted at 2026-06-19 02:53:58 on dist-test-slave-0f9d"
server_key: "6a0915fa4951a2c6e4ee8b0200790ff0"
server_key_iv: "17bfa2d47662ffbab6ab101f509482cd"
server_key_version: "encryptionkey@0"
I20260619 02:53:58.231933 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:58.262550 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:58.265678 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:58.265872 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:58.271270 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39365
I20260619 02:53:58.271314 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39365 every 8 connection(s)
I20260619 02:53:58.272006 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:58.272959 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:58.274788 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15771
I20260619 02:53:58.274979 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:58.275166 27016 external_mini_cluster.cc:1428] Setting key 40233fd0637b88eccec4a1282a5325da
I20260619 02:53:58.276597 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f: Bootstrap starting.
I20260619 02:53:58.277367 15839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:58.277841 15839 log.cc:824] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:58.278710 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f: No bootstrap required, opened a new log
I20260619 02:53:58.281399 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } }
I20260619 02:53:58.281630 15839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:58.281672 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f946f086c084ca283b55091374cfc6f, State: Initialized, Role: FOLLOWER
I20260619 02:53:58.281832 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } }
I20260619 02:53:58.281916 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:58.281960 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:58.282013 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:58.283052 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } }
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:58.283217 15839 leader_election.cc:302] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f946f086c084ca283b55091374cfc6f; no voters:
I20260619 02:53:58.283504 15839 leader_election.cc:288] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:58.283550 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:58.283680 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [term 1 LEADER]: Becoming Leader. State: Replica: 7f946f086c084ca283b55091374cfc6f, State: Running, Role: LEADER
I20260619 02:53:58.283882 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:58.283864 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } }
I20260619 02:53:58.284334 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f946f086c084ca283b55091374cfc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } } }
I20260619 02:53:58.284428 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:58.284677 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f946f086c084ca283b55091374cfc6f. Latest consensus state: current_term: 1 leader_uuid: "7f946f086c084ca283b55091374cfc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f946f086c084ca283b55091374cfc6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39365 } } }
I20260619 02:53:58.284837 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:58.286953 15851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:58.287724 15851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:58.288378 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.275881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:36743 (local address 127.0.0.1:39365)
0619 02:53:58.276181 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.276192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.276226 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:58.276743 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.276753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.276830 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.277010 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.277860 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.279018 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.279843 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.280357 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.280483 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.283995 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:58.284001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:58.284002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:58.284020 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:58.286100 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:58.286796 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:58.286801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:58.286802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:58.286986 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:58.287260 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:58.287261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:58.287261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:58.287622 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:58.287794 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.287998 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.288125 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":138,"threads_started":1}
I20260619 02:53:58.289773 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:58.289898 15851 catalog_manager.cc:1347] Generated new cluster ID: 3ae27e96678349a1bcd150775bc1a2dc
I20260619 02:53:58.289958 15851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:58.301043 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.291857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:53455 (local address 127.0.0.1:39365)
0619 02:53:58.291917 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.291920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.292104 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:53:58.292203 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.292204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.292219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.292290 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.293147 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.293973 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.295288 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.295686 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.295785 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.297638 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:58.297640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:58.297641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:58.297652 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:58.299518 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:58.300134 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:58.300135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:58.300136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:58.300206 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:58.300427 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:58.300427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:58.300428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:58.300578 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:58.300684 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.300846 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.300922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:53:58.305570 15851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:58.306492 15851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:58.311748 15851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f946f086c084ca283b55091374cfc6f: Generated new TSK 0
I20260619 02:53:58.312117 15851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:58.317056 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15771
W20260619 02:53:58.322515 15756 connection.cc:535] client connection to 127.0.0.1:39365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39365): Connection reset by peer (error 104)
2026-06-19T02:53:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15870](info): set up 2 sockets
Jun 19 02:53:58 dist-test-slave-0f9d krb5kdc[15870](info): commencing operation
krb5kdc: starting...
W20260619 02:53:59.303431 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:53:59 dist-test-slave-0f9d krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:59.451356 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:59.550873 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:59.551219 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:59.551256 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:59.552657 15883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:59.552714 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:59.552733 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:59.552752 15883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:59.552767 15883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:59.554417 15883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:53:59.554725 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:59.555011 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:59.560902 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:59.560902 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:59.560935 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:59.561239 15883 server_base.cc:1034] running on GCE node
I20260619 02:53:59.561751 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:59.562134 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:59.563313 15883 hybrid_clock.cc:648] HybridClock initialized: now 1781837639563296 us; error 37 us; skew 500 ppm
Jun 19 02:53:59 dist-test-slave-0f9d krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:59.567296 15883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:59.567946 15883 webserver.cc:460] Webserver started at http://127.0.0.1:42021/ using document root <none> and password file <none>
I20260619 02:53:59.568212 15883 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:59.568269 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:59.568406 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:59.570109 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "990b231f09ec4a1c9a11b402cd1bed29"
format_stamp: "Formatted at 2026-06-19 02:53:59 on dist-test-slave-0f9d"
server_key: "46a3b116bf415c5377e0d73a82e2f9f6"
server_key_iv: "e4f9335ab397eeb7c8d1072a6f1e957b"
server_key_version: "encryptionkey@0"
I20260619 02:53:59.570650 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "990b231f09ec4a1c9a11b402cd1bed29"
format_stamp: "Formatted at 2026-06-19 02:53:59 on dist-test-slave-0f9d"
server_key: "46a3b116bf415c5377e0d73a82e2f9f6"
server_key_iv: "e4f9335ab397eeb7c8d1072a6f1e957b"
server_key_version: "encryptionkey@0"
I20260619 02:53:59.572820 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:59.574231 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:59.574636 15883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:59.574759 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "990b231f09ec4a1c9a11b402cd1bed29"
format_stamp: "Formatted at 2026-06-19 02:53:59 on dist-test-slave-0f9d"
server_key: "46a3b116bf415c5377e0d73a82e2f9f6"
server_key_iv: "e4f9335ab397eeb7c8d1072a6f1e957b"
server_key_version: "encryptionkey@0"
I20260619 02:53:59.574854 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:59.599499 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:59.602195 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:59.602371 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:59.607118 15883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42275
I20260619 02:53:59.607199 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42275 every 8 connection(s)
I20260619 02:53:59.607590 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:59.608623 15951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:59.609076 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15883
I20260619 02:53:59.609198 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:53:59.609382 27016 external_mini_cluster.cc:1428] Setting key 6c899b3c956b76795dcafd10a8c8d3dc
I20260619 02:53:59.612913 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29: Bootstrap starting.
I20260619 02:53:59.613775 15951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:59.614099 15951 log.cc:824] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:59.615278 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29: No bootstrap required, opened a new log
I20260619 02:53:59.616346 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:59.610093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:46127 (local address 127.0.0.1:42275)
0619 02:53:59.610357 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:59.610369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:59.610419 (+ 50us) server_negotiation.cc:407] Connection header received
0619 02:53:59.611096 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:59.611108 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:59.611192 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:59.611390 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:59.611653 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:59.613223 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:59.614286 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:59.614765 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:59.614908 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:59.615684 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:59.615690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:59.615690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:59.615876 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:59.615916 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:59.615930 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:59.616007 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:59.616106 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:53:59.616141 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":122,"threads_started":1}
I20260619 02:53:59.617395 15951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260619 02:53:59.617621 15951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:59.617673 15951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990b231f09ec4a1c9a11b402cd1bed29, State: Initialized, Role: FOLLOWER
I20260619 02:53:59.617816 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:59.617822 15951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260619 02:53:59.617903 15951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:59.617944 15951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:59.618000 15951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:59.618835 15951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260619 02:53:59.619122 15951 leader_election.cc:302] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 990b231f09ec4a1c9a11b402cd1bed29; no voters:
I20260619 02:53:59.619452 15951 leader_election.cc:288] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:59.619527 15960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:59.619724 15960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [term 1 LEADER]: Becoming Leader. State: Replica: 990b231f09ec4a1c9a11b402cd1bed29, State: Running, Role: LEADER
I20260619 02:53:59.619805 15951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:59.619894 15960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260619 02:53:59.620342 15960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990b231f09ec4a1c9a11b402cd1bed29. Latest consensus state: current_term: 1 leader_uuid: "990b231f09ec4a1c9a11b402cd1bed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260619 02:53:59.620435 15960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:59.620643 15962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "990b231f09ec4a1c9a11b402cd1bed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990b231f09ec4a1c9a11b402cd1bed29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260619 02:53:59.620702 15962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:59.621704 15971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:59.622431 15971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:59.624719 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:59.619770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:59783 (local address 127.0.0.1:42275)
0619 02:53:59.619819 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:59.619821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:59.620011 (+ 190us) server_negotiation.cc:407] Connection header received
0619 02:53:59.620124 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:59.620125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:59.620139 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:59.620274 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:59.620506 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:59.622116 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:59.623046 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:59.623277 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:59.623696 (+ 419us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:59.624389 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:59.624391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:59.624391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:59.624399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:59.624420 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:59.624427 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:59.624532 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:59.624620 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 02:53:59.624632 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:53:59.624969 15971 catalog_manager.cc:1347] Generated new cluster ID: 2a6ce15b040741069662b5d17d43c8f8
I20260619 02:53:59.625025 15971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:59.634732 15971 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:59.635377 15971 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:59.638517 15971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 990b231f09ec4a1c9a11b402cd1bed29: Generated new TSK 0
I20260619 02:53:59.638749 15971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:59.647257 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15883
W20260619 02:53:59.652364 15868 connection.cc:535] client connection to 127.0.0.1:42275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42275): Connection reset by peer (error 104)
2026-06-19T02:53:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:59 dist-test-slave-0f9d krb5kdc[15982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:59 dist-test-slave-0f9d krb5kdc[15982](info): set up 2 sockets
Jun 19 02:53:59 dist-test-slave-0f9d krb5kdc[15982](info): commencing operation
krb5kdc: starting...
W20260619 02:54:00.639192 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:00 dist-test-slave-0f9d krb5kdc[15982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:54:00.782500 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:00.885841 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:00.886185 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:00.886216 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:00.888373 15995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:00.888487 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:00.888504 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:00.888517 15995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:00.888530 15995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:00.891124 15995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:00.891557 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:00.891891 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:00.898334 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:00.898344 15995 server_base.cc:1034] running on GCE node
W20260619 02:54:00.898320 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:00.898321 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:00.898998 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:00.899381 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:00.900549 15995 hybrid_clock.cc:648] HybridClock initialized: now 1781837640900507 us; error 72 us; skew 500 ppm
Jun 19 02:54:00 dist-test-slave-0f9d krb5kdc[15982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:00.903831 15995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:54:00.904479 15995 webserver.cc:460] Webserver started at http://127.0.0.1:35439/ using document root <none> and password file <none>
I20260619 02:54:00.904769 15995 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:00.904826 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:00.904958 15995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:00.906280 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "2b2348a5883d4ba789b78222c53776e0"
format_stamp: "Formatted at 2026-06-19 02:54:00 on dist-test-slave-0f9d"
server_key: "456852fb9e5f8c6c52c7b4de66289a1c"
server_key_iv: "e0281e800a3bddf7429af593d82d3911"
server_key_version: "encryptionkey@0"
I20260619 02:54:00.906685 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "2b2348a5883d4ba789b78222c53776e0"
format_stamp: "Formatted at 2026-06-19 02:54:00 on dist-test-slave-0f9d"
server_key: "456852fb9e5f8c6c52c7b4de66289a1c"
server_key_iv: "e0281e800a3bddf7429af593d82d3911"
server_key_version: "encryptionkey@0"
I20260619 02:54:00.908469 15995 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:54:00.909651 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:00.909919 15995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:00.910025 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "2b2348a5883d4ba789b78222c53776e0"
format_stamp: "Formatted at 2026-06-19 02:54:00 on dist-test-slave-0f9d"
server_key: "456852fb9e5f8c6c52c7b4de66289a1c"
server_key_iv: "e0281e800a3bddf7429af593d82d3911"
server_key_version: "encryptionkey@0"
I20260619 02:54:00.910125 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:00.914355 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:00.917035 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:00.917176 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:00.922475 15995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33531
I20260619 02:54:00.922513 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33531 every 8 connection(s)
I20260619 02:54:00.923066 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:00.924252 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:00.927665 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0: Bootstrap starting.
I20260619 02:54:00.928499 16063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:00.928884 16063 log.cc:824] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:00.929626 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 15995
I20260619 02:54:00.929749 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:00.929934 27016 external_mini_cluster.cc:1428] Setting key 6f4278d1b475a64678ed9ef44c02b036
I20260619 02:54:00.930051 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0: No bootstrap required, opened a new log
I20260619 02:54:00.932354 16063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260619 02:54:00.932551 16063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:00.932596 16063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2348a5883d4ba789b78222c53776e0, State: Initialized, Role: FOLLOWER
I20260619 02:54:00.932780 16063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260619 02:54:00.932858 16063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:00.932894 16063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:00.933086 16063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:00.934262 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:00.930568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:41155 (local address 127.0.0.1:33531)
0619 02:54:00.930774 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:00.930786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:00.930820 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:54:00.932313 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:00.932325 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:00.932372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:00.932599 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:00.933685 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:00.933689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:00.933689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:00.933694 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:00.933742 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:00.933756 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:00.933819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:00.933987 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:54:00.934049 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260619 02:54:00.934437 16063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260619 02:54:00.934579 16063 leader_election.cc:302] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b2348a5883d4ba789b78222c53776e0; no voters:
I20260619 02:54:00.934828 16063 leader_election.cc:288] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:00.935125 16069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:00.935222 16063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:00.935276 16069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2b2348a5883d4ba789b78222c53776e0, State: Running, Role: LEADER
I20260619 02:54:00.935388 16069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260619 02:54:00.935858 16069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b2348a5883d4ba789b78222c53776e0. Latest consensus state: current_term: 1 leader_uuid: "2b2348a5883d4ba789b78222c53776e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260619 02:54:00.935863 16068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b2348a5883d4ba789b78222c53776e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2348a5883d4ba789b78222c53776e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260619 02:54:00.936044 16069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:00.936172 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:00.936306 16068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:00.936499 16077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:00.937433 16077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:00.939553 16077 catalog_manager.cc:1347] Generated new cluster ID: f173d4c23ed84194bbb08ca62b0eb8ba
I20260619 02:54:00.939622 16077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:00.940094 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:00.938531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:49225 (local address 127.0.0.1:33531)
0619 02:54:00.938578 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:00.938581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:00.938704 (+ 123us) server_negotiation.cc:407] Connection header received
0619 02:54:00.938780 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:00.938781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:00.938789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:00.938858 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:00.939703 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:00.939704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:00.939705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:00.939710 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:00.939730 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:00.939735 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:00.939783 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:00.939889 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:54:00.939905 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:54:00.967053 16077 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:00.967334 16077 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:00.974212 16077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b2348a5883d4ba789b78222c53776e0: Generated new TSK 0
I20260619 02:54:00.974524 16077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:00.980719 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 15995
2026-06-19T02:54:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:01 dist-test-slave-0f9d krb5kdc[16094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:01 dist-test-slave-0f9d krb5kdc[16094](info): set up 2 sockets
Jun 19 02:54:01 dist-test-slave-0f9d krb5kdc[16094](info): commencing operation
krb5kdc: starting...
W20260619 02:54:01.983649 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:02.138370 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45143
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:02.240036 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:02.240360 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:02.240411 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:02.241935 16107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:02.242018 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:02.242039 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:02.242057 16107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:02.242072 16107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:02.244148 16107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:02.244599 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:02.244880 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:02.251235 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:02.251240 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:02.251235 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:02.251267 16107 server_base.cc:1034] running on GCE node
I20260619 02:54:02.251724 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:02.252097 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:02.253304 16107 hybrid_clock.cc:648] HybridClock initialized: now 1781837642253280 us; error 41 us; skew 500 ppm
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837642, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:02.264204 16107 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:02.264971 16107 webserver.cc:460] Webserver started at http://127.26.98.62:37535/ using document root <none> and password file <none>
I20260619 02:54:02.265286 16107 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:02.265339 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:02.265472 16107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:02.266932 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "e09bc3f9815b4c15bef6db2edb7f1f30"
format_stamp: "Formatted at 2026-06-19 02:54:02 on dist-test-slave-0f9d"
server_key: "c883a5880de8074ce6e37740e5a7974c"
server_key_iv: "8081974fa143c2dfa78e55339f540d04"
server_key_version: "encryptionkey@0"
I20260619 02:54:02.267377 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "e09bc3f9815b4c15bef6db2edb7f1f30"
format_stamp: "Formatted at 2026-06-19 02:54:02 on dist-test-slave-0f9d"
server_key: "c883a5880de8074ce6e37740e5a7974c"
server_key_iv: "8081974fa143c2dfa78e55339f540d04"
server_key_version: "encryptionkey@0"
I20260619 02:54:02.268960 16107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:02.269901 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:02.270264 16107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:02.270371 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "e09bc3f9815b4c15bef6db2edb7f1f30"
format_stamp: "Formatted at 2026-06-19 02:54:02 on dist-test-slave-0f9d"
server_key: "c883a5880de8074ce6e37740e5a7974c"
server_key_iv: "8081974fa143c2dfa78e55339f540d04"
server_key_version: "encryptionkey@0"
I20260619 02:54:02.270465 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:02.284693 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:02.287732 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:02.287925 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:02.293278 16107 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45143
I20260619 02:54:02.293367 16174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45143 every 8 connection(s)
I20260619 02:54:02.293779 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:02.295029 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:02.297170 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16107
I20260619 02:54:02.297405 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:02.297650 27016 external_mini_cluster.cc:1428] Setting key e2a98fa227c22d66ccc95d6acf8dbd66
I20260619 02:54:02.299302 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30: Bootstrap starting.
I20260619 02:54:02.300099 16175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:02.300439 16175 log.cc:824] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:02.301394 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30: No bootstrap required, opened a new log
I20260619 02:54:02.304553 16175 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } }
I20260619 02:54:02.304808 16175 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:02.304838 16175 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09bc3f9815b4c15bef6db2edb7f1f30, State: Initialized, Role: FOLLOWER
I20260619 02:54:02.304993 16175 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } }
I20260619 02:54:02.305075 16175 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:02.305112 16175 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:02.305172 16175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:02.305980 16175 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } }
I20260619 02:54:02.306114 16175 leader_election.cc:302] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09bc3f9815b4c15bef6db2edb7f1f30; no voters:
I20260619 02:54:02.306368 16175 leader_election.cc:288] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:02.306411 16180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:02.306603 16180 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [term 1 LEADER]: Becoming Leader. State: Replica: e09bc3f9815b4c15bef6db2edb7f1f30, State: Running, Role: LEADER
I20260619 02:54:02.306739 16175 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:02.306768 16180 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } }
I20260619 02:54:02.307277 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09bc3f9815b4c15bef6db2edb7f1f30. Latest consensus state: current_term: 1 leader_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } } }
I20260619 02:54:02.307323 16182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09bc3f9815b4c15bef6db2edb7f1f30" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45143 } } }
I20260619 02:54:02.307425 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:02.307430 16182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:02.307663 16187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:02.308888 16187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:02.311282 16187 catalog_manager.cc:1347] Generated new cluster ID: 8913cc7f79b745c6b32cfbf282191ef1
I20260619 02:54:02.311358 16187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:02.312286 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:02.298565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57201 (local address 127.26.98.62:45143)
0619 02:54:02.298827 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:02.298838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:02.298875 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:54:02.299593 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:02.299602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:02.299668 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:02.299850 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:02.300649 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:02.302125 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:02.303146 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:02.304131 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:02.304461 (+ 330us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:02.307622 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:02.307630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:02.307631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:02.307651 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:02.310005 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:02.310667 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:02.310671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:02.310672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:02.310802 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:02.311095 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:02.311096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:02.311097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:02.311469 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:02.311657 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:02.311892 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:54:02.312023 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260619 02:54:02.313400 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:02.317595 16187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:02.318491 16187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:02.324909 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:02.315189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38457 (local address 127.26.98.62:45143)
0619 02:54:02.315237 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:02.315240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:02.315526 (+ 286us) server_negotiation.cc:407] Connection header received
0619 02:54:02.315629 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:02.315630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:02.315645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:02.315707 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:02.316767 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:02.317579 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:02.318850 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:02.319244 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:02.319286 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:02.321061 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:02.321064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:02.321064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:02.321075 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:02.322800 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:02.323539 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:02.323541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:02.323542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:02.323631 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:02.324088 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:02.324089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:02.324089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:02.324260 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:02.324384 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:02.324549 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:54:02.324635 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:54:02.325829 16187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09bc3f9815b4c15bef6db2edb7f1f30: Generated new TSK 0
I20260619 02:54:02.326057 16187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:02.334237 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16107
W20260619 02:54:02.339067 16092 connection.cc:535] client connection to 127.26.98.62:45143 recv error: Network error: failed to read from TLS socket (remote: 127.26.98.62:45143): Connection reset by peer (error 104)
2026-06-19T02:54:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16206](info): set up 2 sockets
Jun 19 02:54:02 dist-test-slave-0f9d krb5kdc[16206](info): commencing operation
krb5kdc: starting...
W20260619 02:54:03.335515 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:03 dist-test-slave-0f9d krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:03.479575 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44473
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:44473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:03.579900 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:03.580228 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:03.580268 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:03.581876 16219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:03.581954 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:03.581974 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:03.581991 16219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:03.582006 16219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:03.584465 16219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:44473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:44473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:03.585150 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:03.585462 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:03.591681 16227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:03.591602 16224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:03.591602 16225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:03.592720 16219 server_base.cc:1034] running on GCE node
I20260619 02:54:03.593102 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:03.593515 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:03.594724 16219 hybrid_clock.cc:648] HybridClock initialized: now 1781837643594676 us; error 69 us; skew 500 ppm
Jun 19 02:54:03 dist-test-slave-0f9d krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837643, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:03.598332 16219 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:03.599222 16219 webserver.cc:460] Webserver started at http://127.26.98.62:35203/ using document root <none> and password file <none>
I20260619 02:54:03.599545 16219 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:03.599608 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:03.599771 16219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:03.601161 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6"
format_stamp: "Formatted at 2026-06-19 02:54:03 on dist-test-slave-0f9d"
server_key: "ec89efe2a63506407729606ad42d408a"
server_key_iv: "2f7017748b62c82c820a701da83619aa"
server_key_version: "encryptionkey@0"
I20260619 02:54:03.601603 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6"
format_stamp: "Formatted at 2026-06-19 02:54:03 on dist-test-slave-0f9d"
server_key: "ec89efe2a63506407729606ad42d408a"
server_key_iv: "2f7017748b62c82c820a701da83619aa"
server_key_version: "encryptionkey@0"
I20260619 02:54:03.603263 16219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 02:54:03.604192 16234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:03.604418 16219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:03.604530 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6"
format_stamp: "Formatted at 2026-06-19 02:54:03 on dist-test-slave-0f9d"
server_key: "ec89efe2a63506407729606ad42d408a"
server_key_iv: "2f7017748b62c82c820a701da83619aa"
server_key_version: "encryptionkey@0"
I20260619 02:54:03.604610 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:03.626467 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:03.629323 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:03.629535 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:03.634755 16219 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:44473
I20260619 02:54:03.635178 16286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:44473 every 8 connection(s)
I20260619 02:54:03.635275 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:03.636133 16287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:03.636709 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16219
I20260619 02:54:03.636827 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:03.637079 27016 external_mini_cluster.cc:1428] Setting key c6a3c5c88c1f2c6a5d034a40fe076aa0
I20260619 02:54:03.639935 16287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: Bootstrap starting.
I20260619 02:54:03.640825 16287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:03.641456 16287 log.cc:824] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:03.642504 16287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: No bootstrap required, opened a new log
I20260619 02:54:03.644399 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:03.637816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54359 (local address 127.26.98.62:44473)
0619 02:54:03.638052 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:03.638063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:03.638120 (+ 57us) server_negotiation.cc:407] Connection header received
0619 02:54:03.638750 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:03.638759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:03.638829 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:03.639068 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:03.639485 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:03.640672 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:03.642106 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:03.642646 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:03.642776 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:03.643657 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:03.643664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:03.643664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:03.643881 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:03.643924 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:03.643938 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:03.644020 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:03.644147 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:54:03.644185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260619 02:54:03.645093 16287 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } }
I20260619 02:54:03.645336 16287 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:03.645381 16287 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2cf46b9dec4e2088f5af4f2ce984f6, State: Initialized, Role: FOLLOWER
I20260619 02:54:03.645529 16287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } }
I20260619 02:54:03.645618 16287 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:03.645676 16287 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:03.645740 16287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:03.645780 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:03.646512 16287 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } }
I20260619 02:54:03.646616 16287 leader_election.cc:302] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2cf46b9dec4e2088f5af4f2ce984f6; no voters:
I20260619 02:54:03.646803 16287 leader_election.cc:288] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:03.646972 16293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:03.647152 16293 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [term 1 LEADER]: Becoming Leader. State: Replica: cd2cf46b9dec4e2088f5af4f2ce984f6, State: Running, Role: LEADER
I20260619 02:54:03.647202 16287 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:03.647254 16293 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } }
I20260619 02:54:03.647689 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2cf46b9dec4e2088f5af4f2ce984f6. Latest consensus state: current_term: 1 leader_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } } }
I20260619 02:54:03.647714 16298 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cf46b9dec4e2088f5af4f2ce984f6" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 44473 } } }
I20260619 02:54:03.647861 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:03.647953 16298 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6 [sys.catalog]: This master's current role is: LEADER
W20260619 02:54:03.649605 16313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:03.649681 16313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:03.649727 16306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:03.649900 16306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:03.651434 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:03.647417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35727 (local address 127.26.98.62:44473)
0619 02:54:03.647462 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:03.647464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:03.647726 (+ 262us) server_negotiation.cc:407] Connection header received
0619 02:54:03.647835 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:03.647836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:03.647849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:03.647912 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:03.648136 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:03.648840 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:03.650005 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:03.650259 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:03.650326 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:03.651096 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:03.651099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:03.651099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:03.651106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:03.651127 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:03.651132 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:03.651186 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:03.651330 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 02:54:03.651345 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:54:03.652119 16306 catalog_manager.cc:1347] Generated new cluster ID: 0ffa3ce00314494bbb0c2139c4c76ba3
I20260619 02:54:03.652335 16306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:03.677121 16306 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:03.677781 16306 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:03.683605 16306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2cf46b9dec4e2088f5af4f2ce984f6: Generated new TSK 0
I20260619 02:54:03.683890 16306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:03.691419 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16219
W20260619 02:54:03.696416 16204 connection.cc:535] client connection to 127.26.98.62:44473 recv error: Network error: failed to read from TLS socket (remote: 127.26.98.62:44473): Connection reset by peer (error 104)
2026-06-19T02:54:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:03 dist-test-slave-0f9d krb5kdc[16318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:03 dist-test-slave-0f9d krb5kdc[16318](info): set up 2 sockets
Jun 19 02:54:03 dist-test-slave-0f9d krb5kdc[16318](info): commencing operation
krb5kdc: starting...
W20260619 02:54:04.687264 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:04 dist-test-slave-0f9d krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:04.825960 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:33023
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:04.925858 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:04.926179 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:04.926224 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:04.928068 16331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:04.928146 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:04.928172 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:04.928190 16331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:04.928206 16331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:04.930727 16331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:33023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:04.931154 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:04.931605 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:04.937698 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:04.937847 16331 server_base.cc:1034] running on GCE node
W20260619 02:54:04.937947 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:04.937698 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:04.938284 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:04.938870 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:04.940141 16331 hybrid_clock.cc:648] HybridClock initialized: now 1781837644940118 us; error 50 us; skew 500 ppm
Jun 19 02:54:04 dist-test-slave-0f9d krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837644, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:04.943415 16331 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:04.944046 16331 webserver.cc:460] Webserver started at http://127.26.98.62:33199/ using document root <none> and password file <none>
I20260619 02:54:04.944288 16331 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:04.944339 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:04.944466 16331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:04.945782 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef"
format_stamp: "Formatted at 2026-06-19 02:54:04 on dist-test-slave-0f9d"
server_key: "7834579cfe6ffb6915c07288c83e45af"
server_key_iv: "eff03c3af9031a2366a77d8fffe64ec8"
server_key_version: "encryptionkey@0"
I20260619 02:54:04.946190 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef"
format_stamp: "Formatted at 2026-06-19 02:54:04 on dist-test-slave-0f9d"
server_key: "7834579cfe6ffb6915c07288c83e45af"
server_key_iv: "eff03c3af9031a2366a77d8fffe64ec8"
server_key_version: "encryptionkey@0"
I20260619 02:54:04.947968 16331 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:54:04.949054 16346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:04.949354 16331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:54:04.949476 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef"
format_stamp: "Formatted at 2026-06-19 02:54:04 on dist-test-slave-0f9d"
server_key: "7834579cfe6ffb6915c07288c83e45af"
server_key_iv: "eff03c3af9031a2366a77d8fffe64ec8"
server_key_version: "encryptionkey@0"
I20260619 02:54:04.949561 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:04.953852 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:04.956480 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:04.956661 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:04.961859 16331 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:33023
I20260619 02:54:04.961908 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:33023 every 8 connection(s)
I20260619 02:54:04.962373 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:04.962726 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16331
I20260619 02:54:04.962869 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:04.963085 27016 external_mini_cluster.cc:1428] Setting key 521e7db6d445d1433fea58a2e2146f85
I20260619 02:54:04.963913 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:04.966180 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:04.963740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43727 (local address 127.26.98.62:33023)
0619 02:54:04.963941 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:04.963950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:04.963992 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:54:04.964561 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:04.964570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:04.964607 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:04.964786 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:04.965671 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:04.965676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:04.965676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:04.965680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:04.965741 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:04.965754 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:04.965816 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:04.965895 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:54:04.965938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260619 02:54:04.967329 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:04.967794 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: Bootstrap starting.
I20260619 02:54:04.968713 16399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:04.969123 16399 log.cc:824] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:04.969937 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: No bootstrap required, opened a new log
I20260619 02:54:04.970173 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:04.969011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41537 (local address 127.26.98.62:33023)
0619 02:54:04.969053 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:04.969055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:04.969101 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:54:04.969181 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:04.969181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:04.969187 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:04.969236 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:04.969923 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:04.969925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:04.969925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:04.969929 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:04.969946 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:04.969951 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:04.970015 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:04.970096 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:54:04.970111 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:54:04.972141 16399 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } }
I20260619 02:54:04.972312 16399 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:04.972335 16399 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49a3a4b7574fd88db8b8e06b0a14ef, State: Initialized, Role: FOLLOWER
I20260619 02:54:04.972450 16399 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } }
I20260619 02:54:04.972537 16399 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:04.972568 16399 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:04.972606 16399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:04.973229 16399 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } }
I20260619 02:54:04.973330 16399 leader_election.cc:302] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce49a3a4b7574fd88db8b8e06b0a14ef; no voters:
I20260619 02:54:04.973487 16399 leader_election.cc:288] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:04.973529 16409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:04.973716 16409 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [term 1 LEADER]: Becoming Leader. State: Replica: ce49a3a4b7574fd88db8b8e06b0a14ef, State: Running, Role: LEADER
I20260619 02:54:04.973759 16399 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:04.973815 16409 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } }
I20260619 02:54:04.974273 16410 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } } }
I20260619 02:54:04.974532 16410 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:04.974466 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce49a3a4b7574fd88db8b8e06b0a14ef. Latest consensus state: current_term: 1 leader_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a3a4b7574fd88db8b8e06b0a14ef" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 33023 } } }
I20260619 02:54:04.974794 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef [sys.catalog]: This master's current role is: LEADER
W20260619 02:54:04.975910 16424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:04.975962 16424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:04.976013 16418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:04.976153 16418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:04.978263 16418 catalog_manager.cc:1347] Generated new cluster ID: 50ae2953f7954fc08905b5211490d9e5
I20260619 02:54:04.978340 16418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:04.990688 16418 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:04.990965 16418 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:04.997047 16418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce49a3a4b7574fd88db8b8e06b0a14ef: Generated new TSK 0
I20260619 02:54:04.997359 16418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:05.009953 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16331
2026-06-19T02:54:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:05 dist-test-slave-0f9d krb5kdc[16430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:05 dist-test-slave-0f9d krb5kdc[16430](info): set up 2 sockets
Jun 19 02:54:05 dist-test-slave-0f9d krb5kdc[16430](info): commencing operation
krb5kdc: starting...
W20260619 02:54:06.019613 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:06.167829 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46221
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:06.268314 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:06.268644 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:06.268674 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:06.270272 16443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:06.270360 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:06.270375 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:06.270387 16443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:06.270399 16443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:06.272536 16443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:46221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:06.272917 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:06.273216 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:06.279204 16448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:06.279213 16451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:06.279204 16449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:06.279878 16443 server_base.cc:1034] running on GCE node
I20260619 02:54:06.280184 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:06.280676 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:06.281910 16443 hybrid_clock.cc:648] HybridClock initialized: now 1781837646281861 us; error 47 us; skew 500 ppm
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837646, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:06.285308 16443 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:06.285930 16443 webserver.cc:460] Webserver started at http://127.26.98.62:42255/ using document root <none> and password file <none>
I20260619 02:54:06.286228 16443 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:06.286288 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:06.286435 16443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:06.288084 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "2f8ddb1c0dac47619ccacfe93deb008e"
format_stamp: "Formatted at 2026-06-19 02:54:06 on dist-test-slave-0f9d"
server_key: "29afa3957881e953d92792883ddb80c2"
server_key_iv: "4a22fd22af98d46cd87f08d68b4bab4e"
server_key_version: "encryptionkey@0"
I20260619 02:54:06.288570 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "2f8ddb1c0dac47619ccacfe93deb008e"
format_stamp: "Formatted at 2026-06-19 02:54:06 on dist-test-slave-0f9d"
server_key: "29afa3957881e953d92792883ddb80c2"
server_key_iv: "4a22fd22af98d46cd87f08d68b4bab4e"
server_key_version: "encryptionkey@0"
I20260619 02:54:06.290344 16443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:54:06.291690 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:06.292048 16443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:54:06.292203 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "2f8ddb1c0dac47619ccacfe93deb008e"
format_stamp: "Formatted at 2026-06-19 02:54:06 on dist-test-slave-0f9d"
server_key: "29afa3957881e953d92792883ddb80c2"
server_key_iv: "4a22fd22af98d46cd87f08d68b4bab4e"
server_key_version: "encryptionkey@0"
I20260619 02:54:06.292309 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:06.306396 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:06.309777 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:06.309978 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:06.315249 16443 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:46221
I20260619 02:54:06.315285 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:46221 every 8 connection(s)
I20260619 02:54:06.315797 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:06.316934 16511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:06.320159 16511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e: Bootstrap starting.
I20260619 02:54:06.320859 16511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:06.321417 16511 log.cc:824] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:06.322685 16511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e: No bootstrap required, opened a new log
I20260619 02:54:06.325330 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16443
I20260619 02:54:06.325515 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:06.325755 27016 external_mini_cluster.cc:1428] Setting key 038589bf52abc379f30db8a217f1aae8
I20260619 02:54:06.325433 16511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } }
I20260619 02:54:06.325868 16511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:06.325927 16511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8ddb1c0dac47619ccacfe93deb008e, State: Initialized, Role: FOLLOWER
I20260619 02:54:06.326154 16511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } }
I20260619 02:54:06.326277 16511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:06.326380 16511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:06.326493 16511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:06.328090 16511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } }
I20260619 02:54:06.328280 16511 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8ddb1c0dac47619ccacfe93deb008e; no voters:
I20260619 02:54:06.328504 16511 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:06.328603 16516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:06.328864 16511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:06.328874 16516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [term 1 LEADER]: Becoming Leader. State: Replica: 2f8ddb1c0dac47619ccacfe93deb008e, State: Running, Role: LEADER
I20260619 02:54:06.329130 16516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } }
I20260619 02:54:06.329711 16518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8ddb1c0dac47619ccacfe93deb008e. Latest consensus state: current_term: 1 leader_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } } }
I20260619 02:54:06.329828 16518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:06.330235 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8ddb1c0dac47619ccacfe93deb008e" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 46221 } } }
I20260619 02:54:06.330314 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:06.330909 16525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:06.331741 16525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:06.334744 16525 catalog_manager.cc:1347] Generated new cluster ID: 915c64cac6564760ae1a380dac58da0c
I20260619 02:54:06.335163 16525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:06.341908 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:06.326419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57235 (local address 127.26.98.62:46221)
0619 02:54:06.326699 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:06.326713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:06.326764 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:54:06.326997 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:06.327007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:06.327088 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:06.327308 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:06.328177 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:06.329748 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:06.331199 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:06.331868 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:06.332037 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:06.336037 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:06.336046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:06.336047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:06.336071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:06.339196 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:06.339915 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:06.339923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:06.339923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:06.340182 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:06.340609 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:06.340610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:06.340611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:06.341108 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:06.341343 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:06.341456 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 02:54:06.341634 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":144,"threads_started":1}
I20260619 02:54:06.346424 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:06.350181 16525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:06.351819 16525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:06.356732 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:06.348471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42387 (local address 127.26.98.62:46221)
0619 02:54:06.348526 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:06.348528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:06.348844 (+ 316us) server_negotiation.cc:407] Connection header received
0619 02:54:06.348996 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:06.348998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:06.349184 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:06.349246 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:06.349986 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:06.350686 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:06.351734 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:06.352136 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:06.352213 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:06.353753 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:06.353755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:06.353756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:06.353767 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:06.355413 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:06.355842 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:06.355843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:06.355844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:06.355922 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:06.356113 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:06.356114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:06.356114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:06.356262 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:06.356379 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:06.356499 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:54:06.356617 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:54:06.362275 16525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8ddb1c0dac47619ccacfe93deb008e: Generated new TSK 0
I20260619 02:54:06.362591 16525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:06.365995 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16443
2026-06-19T02:54:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16542](info): set up 2 sockets
Jun 19 02:54:06 dist-test-slave-0f9d krb5kdc[16542](info): commencing operation
krb5kdc: starting...
W20260619 02:54:07.375595 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:07 dist-test-slave-0f9d krb5kdc[16542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:07.518337 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45209
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:07.619664 16555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:07.619957 16555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:07.619987 16555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:07.621471 16555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:07.621553 16555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:07.621569 16555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:07.621582 16555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:07.621592 16555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:07.623469 16555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:07.623769 16555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:07.624018 16555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:07.630276 16561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:07.630288 16563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:07.630607 16555 server_base.cc:1034] running on GCE node
W20260619 02:54:07.630300 16560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:07.631124 16555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:07.631505 16555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:07.632694 16555 hybrid_clock.cc:648] HybridClock initialized: now 1781837647632675 us; error 35 us; skew 500 ppm
Jun 19 02:54:07 dist-test-slave-0f9d krb5kdc[16542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837647, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:07.639757 16555 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:07.640455 16555 webserver.cc:460] Webserver started at http://127.26.98.62:43711/ using document root <none> and password file <none>
I20260619 02:54:07.640740 16555 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:07.640806 16555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:07.640964 16555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:07.642328 16555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "a2f51e0db7724e229bbf2d2e25a60132"
format_stamp: "Formatted at 2026-06-19 02:54:07 on dist-test-slave-0f9d"
server_key: "ad7358a77ca972180bbab47364ea1062"
server_key_iv: "49abbd8e8ab7401cb772b19631ddcc4a"
server_key_version: "encryptionkey@0"
I20260619 02:54:07.642767 16555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "a2f51e0db7724e229bbf2d2e25a60132"
format_stamp: "Formatted at 2026-06-19 02:54:07 on dist-test-slave-0f9d"
server_key: "ad7358a77ca972180bbab47364ea1062"
server_key_iv: "49abbd8e8ab7401cb772b19631ddcc4a"
server_key_version: "encryptionkey@0"
I20260619 02:54:07.644508 16555 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:54:07.645831 16570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:07.646184 16555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:07.646329 16555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "a2f51e0db7724e229bbf2d2e25a60132"
format_stamp: "Formatted at 2026-06-19 02:54:07 on dist-test-slave-0f9d"
server_key: "ad7358a77ca972180bbab47364ea1062"
server_key_iv: "49abbd8e8ab7401cb772b19631ddcc4a"
server_key_version: "encryptionkey@0"
I20260619 02:54:07.646425 16555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:07.665822 16555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:07.668378 16555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:07.668555 16555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:07.674232 16555 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:45209
I20260619 02:54:07.674319 16622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:45209 every 8 connection(s)
I20260619 02:54:07.674728 16555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:07.675580 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16555
I20260619 02:54:07.675715 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:07.675937 27016 external_mini_cluster.cc:1428] Setting key 8759728d5683583221909e594ec03a48
I20260619 02:54:07.675941 16623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:07.679670 16623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: Bootstrap starting.
I20260619 02:54:07.680573 16623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:07.681372 16623 log.cc:824] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:07.682608 16623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: No bootstrap required, opened a new log
I20260619 02:54:07.683189 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:07.676585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43799 (local address 127.26.98.62:45209)
0619 02:54:07.676799 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:07.676809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:07.676851 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:54:07.677627 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:07.677637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:07.677704 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:07.677866 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:07.678140 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:07.679226 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:07.680163 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:07.680981 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:07.681108 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:07.682025 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:07.682031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:07.682032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:07.682288 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:07.682346 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:07.682367 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:07.682431 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:07.682784 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:54:07.682915 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260619 02:54:07.685117 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:07.686293 16623 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } }
I20260619 02:54:07.686694 16623 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:07.686761 16623 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f51e0db7724e229bbf2d2e25a60132, State: Initialized, Role: FOLLOWER
I20260619 02:54:07.686962 16623 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } }
I20260619 02:54:07.687084 16623 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:07.687126 16623 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:07.687204 16623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:07.688472 16623 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } }
I20260619 02:54:07.688680 16623 leader_election.cc:302] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f51e0db7724e229bbf2d2e25a60132; no voters:
I20260619 02:54:07.688958 16623 leader_election.cc:288] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:07.689072 16633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:07.689299 16633 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [term 1 LEADER]: Becoming Leader. State: Replica: a2f51e0db7724e229bbf2d2e25a60132, State: Running, Role: LEADER
I20260619 02:54:07.689388 16623 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:07.689577 16633 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } }
I20260619 02:54:07.690407 16633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f51e0db7724e229bbf2d2e25a60132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } } }
I20260619 02:54:07.690579 16633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:07.690774 16633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f51e0db7724e229bbf2d2e25a60132. Latest consensus state: current_term: 1 leader_uuid: "a2f51e0db7724e229bbf2d2e25a60132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f51e0db7724e229bbf2d2e25a60132" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 45209 } } }
I20260619 02:54:07.691325 16633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132 [sys.catalog]: This master's current role is: LEADER
W20260619 02:54:07.691625 16648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:07.691700 16648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:07.691767 16645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:07.691998 16645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:07.692478 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:07.687545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40403 (local address 127.26.98.62:45209)
0619 02:54:07.687604 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:07.687607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:07.687690 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:54:07.687786 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:07.687788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:07.687801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:07.687850 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:07.688066 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:07.689015 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:07.690283 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:07.690583 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:07.690654 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:07.691765 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:07.691766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:07.691767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:07.691776 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:07.691796 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:07.691802 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:07.691850 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:07.692163 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:54:07.692200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:54:07.694203 16645 catalog_manager.cc:1347] Generated new cluster ID: bd82a3faccc445028c39ac08d4fa6ac7
I20260619 02:54:07.694278 16645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:07.707165 16645 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:07.707916 16645 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:07.714179 16645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f51e0db7724e229bbf2d2e25a60132: Generated new TSK 0
I20260619 02:54:07.714581 16645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:07.721837 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16555
2026-06-19T02:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:07 dist-test-slave-0f9d krb5kdc[16654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:07 dist-test-slave-0f9d krb5kdc[16654](info): set up 2 sockets
Jun 19 02:54:07 dist-test-slave-0f9d krb5kdc[16654](info): commencing operation
krb5kdc: starting...
W20260619 02:54:08.747404 27016 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:08 dist-test-slave-0f9d krb5kdc[16654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:08.887440 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35149
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:35149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:08.991837 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:08.992175 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:08.992219 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:08.994431 16667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:08.994537 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:08.994565 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:08.994585 16667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:08.994602 16667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:08.997189 16667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:35149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:35149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:08.997547 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:08.997808 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:09.003880 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:09.003881 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:09.004292 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:09.005069 16667 server_base.cc:1034] running on GCE node
I20260619 02:54:09.005640 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:09.006058 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:09.007360 16667 hybrid_clock.cc:648] HybridClock initialized: now 1781837649007325 us; error 58 us; skew 500 ppm
Jun 19 02:54:09 dist-test-slave-0f9d krb5kdc[16654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837649, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:09.011083 16667 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:09.011698 16667 webserver.cc:460] Webserver started at http://127.26.98.62:40895/ using document root <none> and password file <none>
I20260619 02:54:09.011998 16667 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:09.012061 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:09.012236 16667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:09.013612 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "7e0cde3204b34f949545e6a2dfa207c0"
format_stamp: "Formatted at 2026-06-19 02:54:09 on dist-test-slave-0f9d"
server_key: "083946a1b4c8ec076f09830b33612e9c"
server_key_iv: "e8d3dcba5ea19ddfcf2d9bc256ff4166"
server_key_version: "encryptionkey@0"
I20260619 02:54:09.014060 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "7e0cde3204b34f949545e6a2dfa207c0"
format_stamp: "Formatted at 2026-06-19 02:54:09 on dist-test-slave-0f9d"
server_key: "083946a1b4c8ec076f09830b33612e9c"
server_key_iv: "e8d3dcba5ea19ddfcf2d9bc256ff4166"
server_key_version: "encryptionkey@0"
I20260619 02:54:09.016090 16667 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:54:09.017454 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:09.017799 16667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:09.017939 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "7e0cde3204b34f949545e6a2dfa207c0"
format_stamp: "Formatted at 2026-06-19 02:54:09 on dist-test-slave-0f9d"
server_key: "083946a1b4c8ec076f09830b33612e9c"
server_key_iv: "e8d3dcba5ea19ddfcf2d9bc256ff4166"
server_key_version: "encryptionkey@0"
I20260619 02:54:09.018079 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:09.022279 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:09.025437 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:09.025655 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:09.031466 16667 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:35149
I20260619 02:54:09.031520 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:35149 every 8 connection(s)
I20260619 02:54:09.031962 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:09.033346 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:09.035117 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16667
I20260619 02:54:09.035251 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:09.035427 27016 external_mini_cluster.cc:1428] Setting key 22136c8b9ee2c62d4523a921194b04b6
I20260619 02:54:09.036077 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0: Bootstrap starting.
I20260619 02:54:09.036744 16735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:09.037022 16735 log.cc:824] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:09.038148 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0: No bootstrap required, opened a new log
I20260619 02:54:09.039425 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:09.036140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60813 (local address 127.26.98.62:35149)
0619 02:54:09.036371 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:09.036382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:09.036418 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:54:09.037403 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:09.037414 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:09.037451 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:09.037657 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:09.038679 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:09.038685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:09.038686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:09.038692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:09.038758 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:09.038776 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:09.038952 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:09.039089 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:54:09.039145 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260619 02:54:09.040846 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:09.041275 16735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } }
I20260619 02:54:09.041489 16735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:54:09.041531 16735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0cde3204b34f949545e6a2dfa207c0, State: Initialized, Role: FOLLOWER
I20260619 02:54:09.041693 16735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } }
I20260619 02:54:09.041795 16735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:54:09.041849 16735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:54:09.041914 16735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:09.043015 16735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } }
I20260619 02:54:09.043201 16735 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0cde3204b34f949545e6a2dfa207c0; no voters:
I20260619 02:54:09.043397 16735 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:54:09.043717 16745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:09.043758 16735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:09.043882 16745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0cde3204b34f949545e6a2dfa207c0, State: Running, Role: LEADER
I20260619 02:54:09.044001 16745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } }
I20260619 02:54:09.044205 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:09.042552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48143 (local address 127.26.98.62:35149)
0619 02:54:09.042605 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:09.042607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:09.042702 (+ 95us) server_negotiation.cc:407] Connection header received
0619 02:54:09.042790 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:09.042791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:09.042796 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:09.042847 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:09.043714 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:09.043718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:09.043718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:54:09.043727 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:09.043760 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:54:09.043767 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:09.043856 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:09.044075 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:54:09.044113 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:54:09.044484 16745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0cde3204b34f949545e6a2dfa207c0. Latest consensus state: current_term: 1 leader_uuid: "7e0cde3204b34f949545e6a2dfa207c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } } }
I20260619 02:54:09.044588 16745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:09.045734 16751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:09.046419 16751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:09.045348 16746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0cde3204b34f949545e6a2dfa207c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0cde3204b34f949545e6a2dfa207c0" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 35149 } } }
I20260619 02:54:09.047554 16746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:09.050587 16751 catalog_manager.cc:1347] Generated new cluster ID: 7d4d088744864d91808a01f2ccf6dd87
I20260619 02:54:09.050727 16751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:09.090358 16751 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:09.090634 16751 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:09.102264 16751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0cde3204b34f949545e6a2dfa207c0: Generated new TSK 0
I20260619 02:54:09.102577 16751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:54:09.106441 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16667
2026-06-19T02:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1382 ms)
[----------] 18 tests from AuthTokenIssuingTest (24162 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:09 dist-test-slave-0f9d krb5kdc[16766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:09 dist-test-slave-0f9d krb5kdc[16766](info): set up 2 sockets
Jun 19 02:54:09 dist-test-slave-0f9d krb5kdc[16766](info): commencing operation
krb5kdc: starting...
W20260619 02:54:10.123709 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:10.274963 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36823
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:10.374083 16779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:10.374415 16779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:10.374436 16779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:10.374475 16779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:10.376525 16779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:10.376631 16779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:10.376647 16779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:10.376660 16779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:10.376672 16779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:10.379263 16779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:10.379698 16779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:10.380030 16779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:10.386375 16784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:10.386374 16787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:10.386395 16785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:10.386861 16779 server_base.cc:1034] running on GCE node
I20260619 02:54:10.387225 16779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:10.387604 16779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:10.388798 16779 hybrid_clock.cc:648] HybridClock initialized: now 1781837650388757 us; error 69 us; skew 500 ppm
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:10.392880 16779 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:10.393587 16779 webserver.cc:460] Webserver started at http://127.26.98.62:46821/ using document root <none> and password file <none>
I20260619 02:54:10.393841 16779 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:10.393898 16779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:10.394141 16779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:10.395584 16779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "c2251abbc6b8405d8bf0fa2fc7585e16"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "7d9de5c589125c9ca2376b0072c47625"
server_key_iv: "5ec19fa59ee442b612cdadff9aa1e816"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.396013 16779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "c2251abbc6b8405d8bf0fa2fc7585e16"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "7d9de5c589125c9ca2376b0072c47625"
server_key_iv: "5ec19fa59ee442b612cdadff9aa1e816"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.397866 16779 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:54:10.399350 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:10.399784 16779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:10.399950 16779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "c2251abbc6b8405d8bf0fa2fc7585e16"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "7d9de5c589125c9ca2376b0072c47625"
server_key_iv: "5ec19fa59ee442b612cdadff9aa1e816"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.400049 16779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:10.412919 16779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:10.416576 16779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:10.416960 16779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:10.422544 16779 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:36823
I20260619 02:54:10.422581 16846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:36823 every 8 connection(s)
I20260619 02:54:10.423110 16779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:10.424371 16847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:10.426554 16847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:10.432811 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16779
I20260619 02:54:10.432987 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:10.433226 27016 external_mini_cluster.cc:1428] Setting key 57b7cfefa33876b6881d412a58ee5c0f
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:10.440019 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.427548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36823 (local address 127.0.0.1:55350)
0619 02:54:10.427885 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.427963 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.428595 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.428949 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.428953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.429288 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.429499 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.429503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.430676 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.430677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.431625 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.431629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.432591 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.432593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.432699 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.433445 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.433458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.435560 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.437920 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.437932 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.437936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.438268 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.438631 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.438631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.438632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.438737 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.439349 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.439356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.439467 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.439580 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.439775 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"spinlock_wait_cycles":11008,"thread_start_us":158,"threads_started":1}
I20260619 02:54:10.440033 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.427570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.26.98.62:36823)
0619 02:54:10.427897 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.427916 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.428117 (+ 201us) server_negotiation.cc:407] Connection header received
0619 02:54:10.428768 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.428770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.428792 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.428902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.429601 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.430550 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.431742 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.432175 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.432289 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.435695 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.435706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.435706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.435719 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.437781 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.438426 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.438431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.438431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.438549 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.438808 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.438809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.438810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.439239 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.439374 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.439652 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.439906 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
I20260619 02:54:10.441000 16847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:10.441522 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.441349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49692)
0619 02:54:10.441436 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.441491 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 02:54:10.441696 16797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
I20260619 02:54:10.442340 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.442203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49704)
0619 02:54:10.442279 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.442312 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:54:10.442512 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:45063: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.26.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
I20260619 02:54:10.465649 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:45063
--webserver_interface=127.26.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:54:10.491819 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } attempt: 1
I20260619 02:54:10.493178 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.492985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49718)
0619 02:54:10.493082 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.493134 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:54:10.493592 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.493505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49734)
0619 02:54:10.493547 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.493573 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:54:10.493734 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:45063: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
I20260619 02:54:10.561977 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } attempt: 2
I20260619 02:54:10.562825 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.562629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49738)
0619 02:54:10.562736 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.562788 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:54:10.563361 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.563210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49742)
0619 02:54:10.563293 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.563332 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:54:10.563524 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:45063: Network error: Client connection negotiation failed: client connection to 127.26.98.61:45063: connect: Connection refused (error 111)
W20260619 02:54:10.572680 16854 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:10.573009 16854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:10.573032 16854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:10.573063 16854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:10.574690 16854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:10.574790 16854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:10.574806 16854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:10.574826 16854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:10.574839 16854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:10.577131 16854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:10.577543 16854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:10.577834 16854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:10.584432 16859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:10.584412 16862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:10.584412 16860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:10.584823 16854 server_base.cc:1034] running on GCE node
I20260619 02:54:10.585124 16854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:10.585474 16854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:10.586746 16854 hybrid_clock.cc:648] HybridClock initialized: now 1781837650586703 us; error 78 us; skew 500 ppm
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:10.590379 16854 init.cc:377] Logged in from keytab as kudu/127.26.98.61@KRBTEST.COM (short username kudu)
I20260619 02:54:10.591114 16854 webserver.cc:460] Webserver started at http://127.26.98.61:41941/ using document root <none> and password file <none>
I20260619 02:54:10.591409 16854 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:10.591467 16854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:10.591629 16854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:10.593047 16854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data/instance:
uuid: "fed1e371a9244710b03946f11ed6edc5"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "217d648797ab4b6abbde8edf9d933ed2"
server_key_iv: "2772a9f3d56fd64bb6b85956b385d8a0"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.593657 16854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal/instance:
uuid: "fed1e371a9244710b03946f11ed6edc5"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "217d648797ab4b6abbde8edf9d933ed2"
server_key_iv: "2772a9f3d56fd64bb6b85956b385d8a0"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.595489 16854 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 02:54:10.596674 16869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:10.596901 16854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:10.597038 16854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "fed1e371a9244710b03946f11ed6edc5"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "217d648797ab4b6abbde8edf9d933ed2"
server_key_iv: "2772a9f3d56fd64bb6b85956b385d8a0"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.597131 16854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:10.614727 16854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:10.617005 16854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:10.617144 16854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:10.622043 16854 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.61:45063
I20260619 02:54:10.622094 16921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.61:45063 every 8 connection(s)
I20260619 02:54:10.622505 16854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
I20260619 02:54:10.623543 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16854
I20260619 02:54:10.623688 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-1/wal/instance
I20260619 02:54:10.623919 27016 external_mini_cluster.cc:1428] Setting key 0b574eadbd81614091f4a4f5b7b914f8
I20260619 02:54:10.624250 16922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:10.626636 16922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.26.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.60@KRBTEST.COM" created.
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:10.642207 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.627698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.26.98.62:36823)
0619 02:54:10.627782 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.627783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.628299 (+ 516us) server_negotiation.cc:407] Connection header received
0619 02:54:10.629012 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.629014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.629031 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.629088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.629942 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.630488 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.631714 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.631952 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.632016 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.637889 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.637892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.637893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.637905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.640179 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.640843 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.640845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.640845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.640937 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.641255 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.641256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.641256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.641443 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.641529 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.641950 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.642076 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:54:10.642338 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.627673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36823 (local address 127.0.0.1:55362)
0619 02:54:10.628095 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.628151 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.628854 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.629183 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.629188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.629542 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.629821 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.629826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.630598 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.630599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.631605 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.631608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.632046 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.632048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.632211 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.632812 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.632822 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.637646 (+ 4824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.640341 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.640358 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.640363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.640727 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.641022 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.641023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.641024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.641186 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.641540 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.641548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.641765 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.641900 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.642092 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":282,"threads_started":1}
I20260619 02:54:10.643221 16922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:10.652491 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.643677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49758)
0619 02:54:10.643734 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.643748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.643798 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.644108 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.644110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.644274 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.644340 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.644342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.645532 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.645533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.646067 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.646069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.646571 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.646572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.646641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.647177 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.647182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.649353 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.651373 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.651374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.651375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.651544 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.651751 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.651752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.651752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.651809 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.652112 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.652113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.652155 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.652219 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.652348 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:54:10.652501 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.643729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.26.98.61:45063)
0619 02:54:10.643959 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.643964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.643982 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:54:10.644002 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.644004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.644016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.644085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.644406 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.645417 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.646145 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.646502 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.646566 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.649462 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.649479 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.649479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.649487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.651283 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.651619 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.651622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.651622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.651696 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.651857 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.651858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.651858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.652038 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.652154 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.652266 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.652401 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":199,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
I20260619 02:54:10.653342 16922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:10.653852 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.653681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39950)
0619 02:54:10.653736 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.653804 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 02:54:10.654014 16873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
I20260619 02:54:10.654062 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:37383
--webserver_interface=127.26.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:54:10.654692 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } attempt: 3
I20260619 02:54:10.655303 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.655197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39956)
0619 02:54:10.655238 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.655270 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:54:10.655769 16922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:10.664077 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.655065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.26.98.61:45063)
0619 02:54:10.655204 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.655205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.655212 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:54:10.655233 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.655234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.655244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.655293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.655709 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.656478 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.657325 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.657581 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.657644 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.661039 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.661041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.661042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.661051 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.662636 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.663211 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.663213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.663213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.663278 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.663487 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.663488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.663488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.663622 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.663683 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.663904 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.663989 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 02:54:10.664206 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.655018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49760)
0619 02:54:10.655081 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.655092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.655138 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.655313 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.655314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.655515 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.655615 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.655618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.656594 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.656595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.657215 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.657220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.657650 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.657651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.657729 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.659391 (+ 1662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.659398 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.660879 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.662770 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.662771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.662772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.663123 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.663348 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.663349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.663349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.663430 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.663693 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.663694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.663775 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.663857 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.664002 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 02:54:10.664858 16847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:10.665277 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.665162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39960)
0619 02:54:10.665214 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.665247 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:54:10.665920 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.665681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39974)
0619 02:54:10.665823 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.665872 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 02:54:10.666059 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
I20260619 02:54:10.697270 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 1
I20260619 02:54:10.697912 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.697763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39984)
0619 02:54:10.697833 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.697875 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 02:54:10.698495 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.698347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:39998)
0619 02:54:10.698427 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.698467 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 02:54:10.698658 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
I20260619 02:54:10.705070 16922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 1
I20260619 02:54:10.705752 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.705542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40004)
0619 02:54:10.705629 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.705699 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:54:10.706200 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.706096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40020)
0619 02:54:10.706141 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.706174 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:54:10.706362 16922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
W20260619 02:54:10.760286 16929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:10.760607 16929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:10.760628 16929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:10.760653 16929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:10.762197 16929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:10.762280 16929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:10.762295 16929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:10.762307 16929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:10.762318 16929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:10.764518 16929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:36823,127.26.98.61:45063,127.26.98.60:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:10.764909 16929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:10.765237 16929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:10.771353 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:10.771380 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:10.771531 16929 server_base.cc:1034] running on GCE node
W20260619 02:54:10.771351 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:10.771893 16929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:10.772260 16929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:10.773411 16929 hybrid_clock.cc:648] HybridClock initialized: now 1781837650773399 us; error 33 us; skew 500 ppm
I20260619 02:54:10.773873 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 2
I20260619 02:54:10.774402 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.774230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40022)
0619 02:54:10.774293 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.774365 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 02:54:10.774521 16922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 2
I20260619 02:54:10.774803 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.774710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40034)
0619 02:54:10.774746 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.774775 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 02:54:10.774931 16847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
I20260619 02:54:10.774958 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.774787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40040)
0619 02:54:10.774862 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.774922 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:54:10.775296 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.775212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40042)
0619 02:54:10.775252 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.775277 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:54:10.775388 16922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:37383: Network error: Client connection negotiation failed: client connection to 127.26.98.60:37383: connect: Connection refused (error 111)
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:10.776679 16929 init.cc:377] Logged in from keytab as kudu/127.26.98.60@KRBTEST.COM (short username kudu)
I20260619 02:54:10.777242 16929 webserver.cc:460] Webserver started at http://127.26.98.60:44917/ using document root <none> and password file <none>
I20260619 02:54:10.777521 16929 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:10.777578 16929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:10.777725 16929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:10.779213 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data/instance:
uuid: "8fd5aeeb60a44e1baf33e86000162cda"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "2c5b1b8ad2bc030df4bae79e4a59de87"
server_key_iv: "03cd470ed3a3fedc09c7a2e4ac76bee8"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.779635 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal/instance:
uuid: "8fd5aeeb60a44e1baf33e86000162cda"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "2c5b1b8ad2bc030df4bae79e4a59de87"
server_key_iv: "03cd470ed3a3fedc09c7a2e4ac76bee8"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.781445 16929 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:54:10.782554 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:10.782953 16929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:10.783058 16929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "8fd5aeeb60a44e1baf33e86000162cda"
format_stamp: "Formatted at 2026-06-19 02:54:10 on dist-test-slave-0f9d"
server_key: "2c5b1b8ad2bc030df4bae79e4a59de87"
server_key_iv: "03cd470ed3a3fedc09c7a2e4ac76bee8"
server_key_version: "encryptionkey@0"
I20260619 02:54:10.783151 16929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:10.807906 16929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:10.811703 16929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:10.811903 16929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:10.817538 16929 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.60:37383
I20260619 02:54:10.817719 16996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.60:37383 every 8 connection(s)
I20260619 02:54:10.818105 16929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
I20260619 02:54:10.819456 16997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:10.821671 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 16929
I20260619 02:54:10.821808 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837352638110-27016-0/minicluster-data/master-2/wal/instance
I20260619 02:54:10.821599 16997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:10.822028 27016 external_mini_cluster.cc:1428] Setting key 067131a0f8962927de90cdb46073f4ad
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:54:10.833848 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.822920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.26.98.62:36823)
0619 02:54:10.822977 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.822979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.823196 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:54:10.823900 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.823903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.823919 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.823985 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.824715 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.825278 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.826532 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.826866 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.826969 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.830105 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.830108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.830109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.830120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.832204 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.832767 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.832774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.832775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.832847 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.833100 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.833101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.833101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.833242 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.833309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.833596 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.833729 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:54:10.834052 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.822716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.26.98.60:37383)
0619 02:54:10.823098 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.823108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.823136 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:54:10.823725 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.823734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.823760 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.823881 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.824735 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.825847 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.826962 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.827598 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.827684 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.830434 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.830444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.830444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.830453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.832240 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.832757 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.832762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.832762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.832857 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.833065 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.833065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.833066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.833381 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.833569 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.833793 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.833872 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":179,"threads_started":1}
I20260619 02:54:10.834349 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.822796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:36823 (local address 127.0.0.1:55372)
0619 02:54:10.823067 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.823107 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.823757 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.824040 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.824043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.824383 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.824602 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.824607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.825381 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.825382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.826411 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.826415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.827041 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.827043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.827187 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.827946 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.827958 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.829964 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.832395 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.832404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.832407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.832672 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.832922 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.832923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.832923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.833020 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.833314 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.833320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.833386 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.833491 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.833700 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":142,"threads_started":1}
I20260619 02:54:10.835044 16997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:10.843302 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.835335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:45063 (local address 127.0.0.1:49774)
0619 02:54:10.835379 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.835388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.835422 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.835579 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.835580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.835771 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.835835 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.835837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.836508 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.836509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.837055 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.837057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.837434 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.837435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.837517 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.837997 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.838008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.839713 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.842010 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.842013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.842014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.842252 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.842538 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.842540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.842540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.842615 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.842923 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.842924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.842982 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.843044 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.843183 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:54:10.843302 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.835408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.26.98.61:45063)
0619 02:54:10.835461 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.835464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.835472 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:54:10.835497 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.835498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.835510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.835570 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.835913 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.836429 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.837136 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.837370 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.837432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.839843 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.839845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.839846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.839861 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.841840 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.842356 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.842359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.842360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.842445 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.842687 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.842689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.842689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.842836 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.842953 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.843080 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.843183 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:54:10.843971 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.835233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.26.98.62:36823)
0619 02:54:10.835282 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.835285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.835332 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:54:10.835420 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.835421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.835626 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.835680 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.836206 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.836742 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.837464 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.837679 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.837744 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.840313 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.840316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.840316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.840326 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.842422 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.842930 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.842931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.842932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.843007 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.843199 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.843200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.843201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.843328 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.843568 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.843761 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.843866 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:54:10.844080 16997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:10.852145 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.844465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.26.98.60:37383)
0619 02:54:10.844504 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.844506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.844518 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:54:10.844590 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.844592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.844603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.844662 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.844955 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.845540 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.846244 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.846458 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.846516 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.848811 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.848813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.848813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.848821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.850586 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.851254 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.851256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.851256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.851336 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.851539 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.851540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.851541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.851717 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.851785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.851951 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.852039 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:54:10.852479 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.844430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40058)
0619 02:54:10.844479 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.844493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.844542 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.844666 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.844667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.844834 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.844881 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.844884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.845610 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.845611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.846170 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.846172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.846531 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.846532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.846592 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.847196 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.847201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.848672 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.850947 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.850949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.850951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.851169 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.851413 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.851414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.851414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.851471 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.851793 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.851794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.851854 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.851927 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.852139 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":52}
I20260619 02:54:10.853335 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.844773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.26.98.61:45063)
0619 02:54:10.844813 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.844814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.845008 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:54:10.845037 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.845038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.845049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.845107 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.845559 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.846038 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.846958 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.847277 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.847342 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.849852 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.849855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.849855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.849869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.851725 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.852380 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.852382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.852382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.852458 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.852703 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.852704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.852704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.852866 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.853025 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.853103 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.853237 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:54:10.854440 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: Bootstrap starting.
I20260619 02:54:10.855619 16997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:10.855769 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:10.855986 16997 log.cc:824] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:10.857698 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: No bootstrap required, opened a new log
I20260619 02:54:10.858053 16821 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55374
I20260619 02:54:10.859854 16997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.860073 16997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd5aeeb60a44e1baf33e86000162cda, State: Initialized, Role: FOLLOWER
I20260619 02:54:10.860194 16997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.860677 16997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:10.860685 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.860868 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:54:10.863202 17014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:10.863312 17014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:10.866623 16922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 3
I20260619 02:54:10.863689 16929 master_runner.cc:186] Error getting master registration for 127.26.98.62:36823: OK, instance_id { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" instance_seqno: 1781837650420895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:54:10.868309 16929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:54:10.874085 16847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } attempt: 3
I20260619 02:54:10.876631 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.867214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.26.98.60:37383)
0619 02:54:10.867349 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.867352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.867433 (+ 81us) server_negotiation.cc:407] Connection header received
0619 02:54:10.867530 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.867530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.867542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.867611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.867989 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.868961 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.870207 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.870536 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.870605 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.873219 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.873222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.873222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.873233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.875196 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.875665 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.875666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.875667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.875754 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.875971 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.875972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.875972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.876169 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.876236 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.876410 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.876512 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:54:10.876631 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.867154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40072)
0619 02:54:10.867233 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.867338 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.867467 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.867633 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.867634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.867844 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.867905 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.867908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.869096 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.869098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.870081 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.870086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.870611 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.870612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.870675 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.871311 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.871317 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.873064 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.875318 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.875320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.875321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.875586 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.875828 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.875829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.875829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.875909 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.876240 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.876241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.876304 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.876373 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.876513 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
Jun 19 02:54:10 dist-test-slave-0f9d krb5kdc[16766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837650, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:54:10.879303 16922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: Bootstrap starting.
I20260619 02:54:10.880304 16922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:10.880720 16922 log.cc:824] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:10.882393 16922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: No bootstrap required, opened a new log
I20260619 02:54:10.882943 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.874467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:37383 (local address 127.0.0.1:40082)
0619 02:54:10.874543 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:10.874555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:10.874602 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:10.874923 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:10.874924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:10.875086 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:10.875145 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.875148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.876182 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.876183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.877144 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:10.877148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.877637 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.877639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:10.877713 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.878174 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:10.878180 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:10.879567 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:10.881444 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.881446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.881447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.881695 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.882078 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:10.882079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:10.882080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:10.882157 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:10.882524 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:10.882525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:10.882585 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:54:10.882655 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:54:10.882799 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:54:10.882978 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.874545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.26.98.60:37383)
0619 02:54:10.874743 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.874745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.874752 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:54:10.874779 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.874779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.874801 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.874873 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.875241 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.876089 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.877238 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.877562 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.877626 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.879680 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.879683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.879683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.879693 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.881301 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.881795 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.881796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.881797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.881988 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.882229 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.882230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.882231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.882411 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.882479 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.882694 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.882799 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260619 02:54:10.884980 16922 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.885197 16922 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed1e371a9244710b03946f11ed6edc5, State: Initialized, Role: FOLLOWER
I20260619 02:54:10.885250 16847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: Bootstrap starting.
I20260619 02:54:10.885291 16922 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.885764 16922 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:10.885775 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.885987 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.886241 16847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:10.886628 16847 log.cc:824] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:10.887809 16847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: No bootstrap required, opened a new log
W20260619 02:54:10.888638 17030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:10.889082 17030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:10.890183 16847 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.890403 16847 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2251abbc6b8405d8bf0fa2fc7585e16, State: Initialized, Role: FOLLOWER
I20260619 02:54:10.890511 16847 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.891148 16847 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:10.891206 16821 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:54:10.891264 16821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:10.892467 16821 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.888615 16854 master_runner.cc:186] Error getting master registration for 127.26.98.62:36823: OK, instance_id { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" instance_seqno: 1781837650420895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:54:10.892805 16854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:54:10.892845 16821 leader_election.cc:288] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [CANDIDATE]: Term 1 election: Requested vote from peers fed1e371a9244710b03946f11ed6edc5 (127.26.98.61:45063), 8fd5aeeb60a44e1baf33e86000162cda (127.26.98.60:37383)
I20260619 02:54:10.893332 16971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8fd5aeeb60a44e1baf33e86000162cda"
I20260619 02:54:10.893527 16971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:54:10.893752 17043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:10.893798 17043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:10.894124 17032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.894198 17032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.894738 16971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2251abbc6b8405d8bf0fa2fc7585e16 in term 1.
I20260619 02:54:10.896814 16798 leader_election.cc:302] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fd5aeeb60a44e1baf33e86000162cda, c2251abbc6b8405d8bf0fa2fc7585e16; no voters:
I20260619 02:54:10.897733 16896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fed1e371a9244710b03946f11ed6edc5"
I20260619 02:54:10.897902 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:10.899192 16896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2251abbc6b8405d8bf0fa2fc7585e16 in term 1.
I20260619 02:54:10.902801 16779 master_runner.cc:263] No leader master found from master c2251abbc6b8405d8bf0fa2fc7585e16
I20260619 02:54:10.902905 16779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:54:10.903081 17032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:10.903247 17032 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [term 1 LEADER]: Becoming Leader. State: Replica: c2251abbc6b8405d8bf0fa2fc7585e16, State: Running, Role: LEADER
I20260619 02:54:10.903375 17032 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } }
I20260619 02:54:10.904058 17032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2251abbc6b8405d8bf0fa2fc7585e16. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.904147 17032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:10.908025 17054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:10.908339 17054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 02:54:10.918078 16800 tablet.cc:2369] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:54:10.920167 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.900258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.26.98.61:45063)
0619 02:54:10.900314 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.900317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.904844 (+ 4527us) server_negotiation.cc:407] Connection header received
0619 02:54:10.904990 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.904992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.905007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.905044 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.906017 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.906804 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.909541 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.909891 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.909952 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.915111 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.915114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.915115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.915126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.916842 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.918559 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.918561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.918562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.918647 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.918873 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.918874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.918874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.919121 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.919245 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.919959 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.920055 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:54:10.923861 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.901598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.26.98.60:37383)
0619 02:54:10.901650 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.901652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.908343 (+ 6691us) server_negotiation.cc:407] Connection header received
0619 02:54:10.908570 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.908571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.908588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.908627 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.910636 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.911499 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.914037 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.914405 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.914452 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.920684 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.920686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.920686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.920698 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.922431 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.922834 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.922835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.922836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.922941 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.923149 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.923150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.923150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.923306 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.923428 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.923655 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.923750 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:54:10.924346 16971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [term 1 FOLLOWER]: Refusing update from remote peer c2251abbc6b8405d8bf0fa2fc7585e16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:54:10.924811 17052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:54:10.925087 16896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [term 1 FOLLOWER]: Refusing update from remote peer c2251abbc6b8405d8bf0fa2fc7585e16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:54:10.927086 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2251abbc6b8405d8bf0fa2fc7585e16. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.927201 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.927377 17052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:54:10.927754 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:10.900496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.26.98.62:36823)
0619 02:54:10.900539 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:10.900541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:10.908159 (+ 7618us) server_negotiation.cc:407] Connection header received
0619 02:54:10.908515 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:10.908517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:10.908532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:10.908579 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:10.919588 (+ 11009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.920306 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.922209 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:10.922552 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:10.922599 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:10.924032 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:10.924034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:10.924035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:10.924044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:10.926294 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.926746 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.926748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.926748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.926812 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:10.927057 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:10.927058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:10.927058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:10.927217 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:10.927306 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:10.927564 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 02:54:10.927659 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:54:10.929080 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2251abbc6b8405d8bf0fa2fc7585e16. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.929201 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.931800 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.931998 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:10.932722 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.932828 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.933270 17058 mvcc.cc:205] Tried to move back new op lower bound from 7298407018182328320 to 7298407018101493760. Current Snapshot: MvccSnapshot[applied={T|T < 7298407018182328320}]
I20260619 02:54:10.933699 17032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.933802 17032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16 [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:10.934356 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2251abbc6b8405d8bf0fa2fc7585e16" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 36823 } } peers { permanent_uuid: "fed1e371a9244710b03946f11ed6edc5" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 45063 } } peers { permanent_uuid: "8fd5aeeb60a44e1baf33e86000162cda" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 37383 } } }
I20260619 02:54:10.934463 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:10.935704 17056 mvcc.cc:205] Tried to move back new op lower bound from 7298407018182328320 to 7298407018101493760. Current Snapshot: MvccSnapshot[applied={T|T < 7298407018182328320}]
I20260619 02:54:10.936918 17054 catalog_manager.cc:1347] Generated new cluster ID: 316d96194e4e49419ca8c90e33e473f4
I20260619 02:54:10.936995 17054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:10.953514 17054 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:10.954449 17054 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:10.974469 17054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2251abbc6b8405d8bf0fa2fc7585e16: Generated new TSK 0
I20260619 02:54:10.974866 17054 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:54:11.006122 16850 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.003971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.26.98.62:36823)
0619 02:54:11.004031 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.004033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.005245 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 02:54:11.005372 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.005374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.005389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.005420 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.006018 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55390: BlockingRecv error: recv got EOF from 127.0.0.1:55390 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 02:54:11.007077 17016 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.004453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.26.98.60:37383)
0619 02:54:11.004517 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.004519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.006428 (+ 1909us) server_negotiation.cc:407] Connection header received
0619 02:54:11.006551 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.006552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.006566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.006597 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.006996 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40096: BlockingRecv error: recv got EOF from 127.0.0.1:40096 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 02:54:11.007939 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.004880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.26.98.61:45063)
0619 02:54:11.004930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.004932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.007360 (+ 2428us) server_negotiation.cc:407] Connection header received
0619 02:54:11.007459 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.007461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.007485 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.007516 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.007866 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49812: BlockingRecv error: recv got EOF from 127.0.0.1:49812 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 02:54:11.011478 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.010288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.26.98.61:45063)
0619 02:54:11.010359 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.010361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.010671 (+ 310us) server_negotiation.cc:407] Connection header received
0619 02:54:11.010789 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.010791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.010816 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.010847 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.011404 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49814: BlockingRecv error: recv got EOF from 127.0.0.1:49814 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 02:54:11.016012 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.014658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.26.98.61:45063)
0619 02:54:11.014718 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.014720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.015205 (+ 485us) server_negotiation.cc:407] Connection header received
0619 02:54:11.015349 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.015350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.015368 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.015404 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.015929 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49828: BlockingRecv error: recv got EOF from 127.0.0.1:49828 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 02:54:11.021167 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.019848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.26.98.61:45063)
0619 02:54:11.019907 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.019910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.020300 (+ 390us) server_negotiation.cc:407] Connection header received
0619 02:54:11.020503 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.020505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.020528 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.020565 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.021087 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49834: BlockingRecv error: recv got EOF from 127.0.0.1:49834 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 02:54:11.028640 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.027173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.26.98.61:45063)
0619 02:54:11.027233 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.027236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.027674 (+ 438us) server_negotiation.cc:407] Connection header received
0619 02:54:11.027805 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.027807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.027822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.027963 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.028550 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49846: BlockingRecv error: recv got EOF from 127.0.0.1:49846 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 02:54:11.039779 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.038483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.26.98.61:45063)
0619 02:54:11.038547 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.038549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.039036 (+ 487us) server_negotiation.cc:407] Connection header received
0619 02:54:11.039161 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.039162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.039177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.039210 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.039697 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49852: BlockingRecv error: recv got EOF from 127.0.0.1:49852 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 02:54:11.059303 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.057860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.26.98.61:45063)
0619 02:54:11.057929 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.057931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.058416 (+ 485us) server_negotiation.cc:407] Connection header received
0619 02:54:11.058558 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.058559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.058576 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.058623 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.059215 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49862: BlockingRecv error: recv got EOF from 127.0.0.1:49862 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 02:54:11.096408 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.095148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.26.98.61:45063)
0619 02:54:11.095210 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.095212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.095625 (+ 413us) server_negotiation.cc:407] Connection header received
0619 02:54:11.095746 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.095748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.095764 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.095799 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.096326 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49872: BlockingRecv error: recv got EOF from 127.0.0.1:49872 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 02:54:11.163636 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.162243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.26.98.61:45063)
0619 02:54:11.162318 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.162321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.162508 (+ 187us) server_negotiation.cc:407] Connection header received
0619 02:54:11.162547 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.162548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.162586 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.162642 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.163419 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49886: BlockingRecv error: recv got EOF from 127.0.0.1:49886 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 02:54:11.293817 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.292908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.26.98.61:45063)
0619 02:54:11.292986 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.292988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.293002 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:54:11.293110 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.293111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.293124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.293179 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.293729 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49902: BlockingRecv error: recv got EOF from 127.0.0.1:49902 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 02:54:11.552119 16927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:11.551405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.26.98.61:45063)
0619 02:54:11.551462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:11.551465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:11.551562 (+ 97us) server_negotiation.cc:407] Connection header received
0619 02:54:11.551600 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:11.551601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:11.551615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:11.551673 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:11.552044 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49916: BlockingRecv error: recv got EOF from 127.0.0.1:49916 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:54:11.863915 17014 catalog_manager.cc:1259] Loaded cluster ID: 316d96194e4e49419ca8c90e33e473f4
I20260619 02:54:11.864033 17014 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: loading cluster ID for follower catalog manager: success
I20260619 02:54:11.864974 17014 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: acquiring CA information for follower catalog manager: success
I20260619 02:54:11.865334 17014 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fd5aeeb60a44e1baf33e86000162cda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:54:11.890338 17030 catalog_manager.cc:1259] Loaded cluster ID: 316d96194e4e49419ca8c90e33e473f4
I20260619 02:54:11.890592 17030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: loading cluster ID for follower catalog manager: success
I20260619 02:54:11.891686 17030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: acquiring CA information for follower catalog manager: success
I20260619 02:54:11.892294 17030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fed1e371a9244710b03946f11ed6edc5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:54:12.069700 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:12.065774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.26.98.61:45063)
0619 02:54:12.065950 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:12.065952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:12.065967 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:54:12.066009 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:12.066010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:12.066085 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:12.066219 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:12.066542 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.067196 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.068535 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.068774 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.068863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:12.069094 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:12.069257 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:12.069329 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:12.069545 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 02:54:12.069597 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260619 02:54:12.075439 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:12.072457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.26.98.60:37383)
0619 02:54:12.072680 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:12.072682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:12.072690 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:54:12.072738 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:12.072739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:12.072787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:12.072880 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:12.073044 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.073589 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.074473 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.074697 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.074762 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:12.074995 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:12.075157 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:12.075247 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:12.075286 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 02:54:12.075321 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260619 02:54:12.081341 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:12.078260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.26.98.60:37383)
0619 02:54:12.078321 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:12.078323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:12.078490 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:54:12.078584 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:12.078585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:12.078599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:12.078645 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:12.078981 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.079485 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.080493 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.080801 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.080874 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:12.081035 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:12.081077 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:12.081127 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:12.081200 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:54:12.081229 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:54:12.081700 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:12.077976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.26.98.61:45063)
0619 02:54:12.078037 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:12.078039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:12.078430 (+ 391us) server_negotiation.cc:407] Connection header received
0619 02:54:12.078506 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:12.078507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:12.078521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:12.078572 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:12.078772 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.079528 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.080691 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.080961 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.081029 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:12.081242 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:12.081283 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:12.081333 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:12.081577 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:54:12.081614 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:54:12.082620 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:12.078065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.26.98.62:36823)
0619 02:54:12.078329 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:12.078331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:12.078343 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:54:12.078407 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:12.078408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:12.078483 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:12.078621 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:12.079596 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.080509 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.081528 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:12.081851 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:12.082058 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:12.082142 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:12.082257 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:12.082300 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:12.082398 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 02:54:12.082426 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":198,"threads_started":1}
I20260619 02:54:12.085736 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16779
I20260619 02:54:12.092334 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16854
I20260619 02:54:12.100561 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 16929
2026-06-19T02:54:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2996 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:54:12 dist-test-slave-0f9d krb5kdc[17145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:54:12 dist-test-slave-0f9d krb5kdc[17145](info): set up 2 sockets
Jun 19 02:54:12 dist-test-slave-0f9d krb5kdc[17145](info): commencing operation
krb5kdc: starting...
W20260619 02:54:13.122152 27016 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/test-user.keytab.
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
Entry for principal kudu/127.26.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab.
I20260619 02:54:13.288414 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39853
--webserver_interface=127.26.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:54:13.385798 17158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:13.386132 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:13.386153 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:13.386180 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:13.388852 17158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:13.388983 17158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:13.389000 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:13.389014 17158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:13.389025 17158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:13.391530 17158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.62.keytab
--principal=kudu/127.26.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.62:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:13.391886 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:13.392167 17158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:13.398595 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.398664 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.398695 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:13.399111 17158 server_base.cc:1034] running on GCE node
I20260619 02:54:13.399458 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:13.399849 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:13.401050 17158 hybrid_clock.cc:648] HybridClock initialized: now 1781837653401032 us; error 38 us; skew 500 ppm
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:13.411118 17158 init.cc:377] Logged in from keytab as kudu/127.26.98.62@KRBTEST.COM (short username kudu)
I20260619 02:54:13.411933 17158 webserver.cc:460] Webserver started at http://127.26.98.62:43401/ using document root <none> and password file <none>
I20260619 02:54:13.412285 17158 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:13.412351 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:13.412550 17158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:13.414289 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data/instance:
uuid: "5617d6582b414c62bf0ae970791d238f"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "4351501c91fb0312861da22c5246e4c9"
server_key_iv: "ba05895ba3472d4aa88bcc0586e5a5d4"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.414767 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance:
uuid: "5617d6582b414c62bf0ae970791d238f"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "4351501c91fb0312861da22c5246e4c9"
server_key_iv: "ba05895ba3472d4aa88bcc0586e5a5d4"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.417033 17158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 02:54:13.418349 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:13.418709 17158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:13.418920 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal
uuid: "5617d6582b414c62bf0ae970791d238f"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "4351501c91fb0312861da22c5246e4c9"
server_key_iv: "ba05895ba3472d4aa88bcc0586e5a5d4"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.419027 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:13.427901 17158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:13.431341 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:13.431560 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:13.439013 17158 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.62:39853
I20260619 02:54:13.439067 17225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.62:39853 every 8 connection(s)
I20260619 02:54:13.439604 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/data/info.pb
I20260619 02:54:13.441186 17226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:13.443334 17226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:13.446007 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 17158
I20260619 02:54:13.446132 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-0/wal/instance
I20260619 02:54:13.446390 27016 external_mini_cluster.cc:1428] Setting key 697b7a36bbd12938ac378806786ccee3
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.98.61@KRBTEST.COM; defaulting to no policy
I20260619 02:54:13.457350 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.444629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.26.98.62:39853)
0619 02:54:13.444767 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.444772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.444805 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:54:13.445546 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.445548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.445573 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.445677 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.446625 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.447784 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.448893 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.449253 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.449379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.452490 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.452500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.452501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.452511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.455069 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.455852 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.455858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.455858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.455995 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.456247 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.456248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.456248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.456563 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.456668 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.456880 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.457030 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260619 02:54:13.457347 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.444348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39853 (local address 127.0.0.1:55416)
0619 02:54:13.444671 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.444721 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.445339 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.445712 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.445714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.446203 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.446502 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.446510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.447959 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.447960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.448794 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.448796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.449370 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.449371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.449445 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.450087 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.450098 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.452336 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.455252 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.455272 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.455277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.455681 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.456068 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.456068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.456069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.456174 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.456642 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.456648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.456725 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.456824 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.457116 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":103,"thread_start_us":151,"threads_started":1}
I20260619 02:54:13.459128 17226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:13.459607 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.459466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37108)
0619 02:54:13.459511 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.459572 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260619 02:54:13.459791 17177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
I20260619 02:54:13.460417 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.460287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37110)
0619 02:54:13.460347 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.460386 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.61@KRBTEST.COM" created.
W20260619 02:54:13.461717 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:39291: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
Entry for principal kudu/127.26.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab.
I20260619 02:54:13.482453 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:39291
--webserver_interface=127.26.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:54:13.511057 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } attempt: 1
I20260619 02:54:13.511770 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.511596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37116)
0619 02:54:13.511669 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.511728 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 02:54:13.512240 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.512146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37126)
0619 02:54:13.512188 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.512220 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:54:13.512346 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:39291: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
I20260619 02:54:13.580555 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } attempt: 2
I20260619 02:54:13.581283 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.581066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37136)
0619 02:54:13.581177 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.581236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 02:54:13.581728 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.581618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37150)
0619 02:54:13.581667 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.581702 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:54:13.581871 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.61:39291: Network error: Client connection negotiation failed: client connection to 127.26.98.61:39291: connect: Connection refused (error 111)
W20260619 02:54:13.584820 17233 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:13.585176 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:13.585197 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:13.585225 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:13.586863 17233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:13.586994 17233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:13.587013 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:13.587025 17233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:13.587037 17233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:13.589237 17233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.61.keytab
--principal=kudu/127.26.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.61:39291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:13.589618 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:13.589912 17233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:13.595840 17241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.595872 17239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.596002 17238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:13.596629 17233 server_base.cc:1034] running on GCE node
I20260619 02:54:13.596987 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:13.597363 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:13.598565 17233 hybrid_clock.cc:648] HybridClock initialized: now 1781837653598530 us; error 53 us; skew 500 ppm
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:13.602182 17233 init.cc:377] Logged in from keytab as kudu/127.26.98.61@KRBTEST.COM (short username kudu)
I20260619 02:54:13.602845 17233 webserver.cc:460] Webserver started at http://127.26.98.61:42205/ using document root <none> and password file <none>
I20260619 02:54:13.603175 17233 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:13.603241 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:13.603402 17233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:13.605130 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data/instance:
uuid: "b29f8f8d72474603b9de0e7a79870937"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a374628638cf17e028458ac68ddc128"
server_key_iv: "308a67b74422fd711cc5df04f037805e"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.605633 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal/instance:
uuid: "b29f8f8d72474603b9de0e7a79870937"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a374628638cf17e028458ac68ddc128"
server_key_iv: "308a67b74422fd711cc5df04f037805e"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.607380 17233 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 02:54:13.608471 17248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:13.608829 17233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:54:13.608960 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal
uuid: "b29f8f8d72474603b9de0e7a79870937"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a374628638cf17e028458ac68ddc128"
server_key_iv: "308a67b74422fd711cc5df04f037805e"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.609091 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:13.628495 17233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:13.631939 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:13.632195 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:13.638432 17233 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.61:39291
I20260619 02:54:13.638501 17300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.61:39291 every 8 connection(s)
I20260619 02:54:13.638999 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/data/info.pb
I20260619 02:54:13.640067 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 17233
I20260619 02:54:13.640200 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-1/wal/instance
I20260619 02:54:13.640441 27016 external_mini_cluster.cc:1428] Setting key b01d6c0249a6db5428ae728642f7eb02
I20260619 02:54:13.640481 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:13.642768 17301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.26.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.98.60@KRBTEST.COM" created.
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:13.659273 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.643754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.26.98.62:39853)
0619 02:54:13.643836 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.643839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.644304 (+ 465us) server_negotiation.cc:407] Connection header received
0619 02:54:13.645063 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.645065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.645081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.645168 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.646036 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.646946 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.648311 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.648672 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.648758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.655186 (+ 6428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.655188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.655189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.655200 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.657467 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.658083 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.658084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.658085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.658153 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.658446 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.658447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.658448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.658596 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.658661 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.659038 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.659159 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:54:13.659523 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.643768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39853 (local address 127.0.0.1:55426)
0619 02:54:13.644123 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.644180 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.644927 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.645257 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.645263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.645615 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.645889 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.645895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.647107 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.647109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.648189 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.648193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.648823 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.648825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.648996 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.649621 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.649632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.654970 (+ 5338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.657626 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.657644 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.657648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.657996 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.658215 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.658216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.658216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.658372 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.658661 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.658670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.658852 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.658999 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.659225 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":218,"threads_started":1}
I20260619 02:54:13.660264 17301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:13.672596 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.660727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37166)
0619 02:54:13.660813 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.660827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.660883 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.661242 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.661243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.661454 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.661580 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.661583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.662605 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.662607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.663503 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.663509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.664283 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.664284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.664360 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.664905 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.664910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.667177 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.670248 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.670250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.670251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.670567 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.671138 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.671140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.671140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.671325 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.672021 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.672022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.672196 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.672310 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.672486 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:54:13.672739 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.660829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.26.98.61:39291)
0619 02:54:13.661065 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.661071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.661093 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:54:13.661111 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.661113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.661127 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.661207 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.661680 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.662474 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.663721 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.664178 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.664258 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.667510 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.667531 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.667531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.667544 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.670058 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.670814 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.670821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.670821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.670995 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.671449 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.671451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.671451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.671854 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.672189 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.672384 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.672606 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260619 02:54:13.673039 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } attempt: 3
I20260619 02:54:13.673568 17301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:13.674005 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.673848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35446)
0619 02:54:13.673912 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.673973 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
Entry for principal kudu/127.26.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab.
W20260619 02:54:13.674337 17252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
I20260619 02:54:13.674964 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.674819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35462)
0619 02:54:13.674871 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.674926 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 02:54:13.675117 17301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
I20260619 02:54:13.675485 27016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcBfLLw/build/release/bin/kudu
/tmp/dist-test-taskcBfLLw/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:40753
--webserver_interface=127.26.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.98.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:13.682992 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.673849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.26.98.61:39291)
0619 02:54:13.673889 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.673891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.674028 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:54:13.674054 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.674055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.674069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.674169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.674588 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.675406 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.676511 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.676882 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.676969 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.679840 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.679843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.679843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.679856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.681439 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.681951 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.681953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.681953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.682022 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.682319 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.682320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.682320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.682501 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.682567 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.682745 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.682844 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:54:13.683003 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.673827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37176)
0619 02:54:13.673916 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.673928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.673995 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.674192 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.674193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.674400 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.674495 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.674498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.675560 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.675564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.676363 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.676368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.676969 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.676971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.677068 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.677615 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.677621 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.679662 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.681561 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.681563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.681564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.681846 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.682094 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.682095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.682095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.682231 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.682573 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.682574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.682646 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.682725 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.682860 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:54:13.683533 17226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:13.683980 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.683848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35466)
0619 02:54:13.683901 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.683945 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:54:13.684366 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.684261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35474)
0619 02:54:13.684308 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.684345 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:54:13.684453 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
I20260619 02:54:13.715647 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 1
I20260619 02:54:13.716418 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.716191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35484)
0619 02:54:13.716286 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.716370 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:54:13.717015 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.716866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35488)
0619 02:54:13.716925 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.716985 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:54:13.717195 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
I20260619 02:54:13.724449 17301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 1
I20260619 02:54:13.725142 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.724958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35500)
0619 02:54:13.725048 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.725104 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:54:13.725548 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.725459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35512)
0619 02:54:13.725503 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.725530 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:54:13.725641 17301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
W20260619 02:54:13.777750 17308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:54:13.778118 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:54:13.778151 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:54:13.778190 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:54:13.779963 17308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:54:13.780057 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:54:13.780093 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:54:13.780114 17308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:54:13.780133 17308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:54:13.783324 17308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.98.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.98.62:39853,127.26.98.61:39291,127.26.98.60:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/krb5kdc/kudu_127.26.98.60.keytab
--principal=kudu/127.26.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.98.60:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:53:23 UTC on ccycloud.quasar-wqwdzy.root.comops.site
build id 4454976
I20260619 02:54:13.783767 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:54:13.784086 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:54:13.790086 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.790068 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:54:13.790067 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:54:13.790182 17308 server_base.cc:1034] running on GCE node
I20260619 02:54:13.790611 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:54:13.791126 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:54:13.792304 17308 hybrid_clock.cc:648] HybridClock initialized: now 1781837653792287 us; error 38 us; skew 500 ppm
I20260619 02:54:13.792425 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 2
I20260619 02:54:13.792955 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.792736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35526)
0619 02:54:13.792808 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.792908 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:54:13.793360 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.793267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35532)
0619 02:54:13.793314 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.793342 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:54:13.793457 17226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
I20260619 02:54:13.793800 17301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 2
I20260619 02:54:13.794279 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.794183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35548)
0619 02:54:13.794224 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.794256 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 02:54:13.794632 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.794547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35556)
0619 02:54:13.794585 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.794610 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:54:13.794718 17301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.98.60:40753: Network error: Client connection negotiation failed: client connection to 127.26.98.60:40753: connect: Connection refused (error 111)
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:54:13.795722 17308 init.cc:377] Logged in from keytab as kudu/127.26.98.60@KRBTEST.COM (short username kudu)
I20260619 02:54:13.796478 17308 webserver.cc:460] Webserver started at http://127.26.98.60:42205/ using document root <none> and password file <none>
I20260619 02:54:13.796741 17308 fs_manager.cc:359] Metadata directory not provided
I20260619 02:54:13.796795 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:54:13.796911 17308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:54:13.798646 17308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data/instance:
uuid: "2d054782c8624902b282e42ba369f3fb"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a8e1c76f8c8e417a3e8e0ab460273cb"
server_key_iv: "49d36b7ffcd970d7eb0c3cf0139e417b"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.799185 17308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal/instance:
uuid: "2d054782c8624902b282e42ba369f3fb"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a8e1c76f8c8e417a3e8e0ab460273cb"
server_key_iv: "49d36b7ffcd970d7eb0c3cf0139e417b"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.800979 17308 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:54:13.802215 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:54:13.802595 17308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:54:13.802726 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data,/tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal
uuid: "2d054782c8624902b282e42ba369f3fb"
format_stamp: "Formatted at 2026-06-19 02:54:13 on dist-test-slave-0f9d"
server_key: "9a8e1c76f8c8e417a3e8e0ab460273cb"
server_key_iv: "49d36b7ffcd970d7eb0c3cf0139e417b"
server_key_version: "encryptionkey@0"
I20260619 02:54:13.802851 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:54:13.823593 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:54:13.826653 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:54:13.826839 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:54:13.832219 17308 rpc_server.cc:307] RPC server started. Bound to: 127.26.98.60:40753
I20260619 02:54:13.832612 17375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.98.60:40753 every 8 connection(s)
I20260619 02:54:13.832717 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/data/info.pb
I20260619 02:54:13.833351 27016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcBfLLw/build/release/bin/kudu as pid 17308
I20260619 02:54:13.833535 27016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcBfLLw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837352638110-27016-0/minicluster-data/master-2/wal/instance
I20260619 02:54:13.833840 27016 external_mini_cluster.cc:1428] Setting key b0a4365cd2e2ce3d89c2ca816c2859e1
I20260619 02:54:13.834288 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:54:13.837303 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:13.847226 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.834538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.26.98.60:40753)
0619 02:54:13.834755 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.834764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.834796 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:54:13.835682 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.835694 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.835766 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.835945 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.836683 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.837776 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.838922 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.839529 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.839649 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.842800 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.842810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.842810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.842817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.844852 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.845251 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.845254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.845255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.845358 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.845658 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.845661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.845662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.846169 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.846328 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.846657 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.846924 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260619 02:54:13.848168 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.838174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.62:39853 (local address 127.0.0.1:55442)
0619 02:54:13.838470 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.838561 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.838712 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.839079 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.839082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.839437 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.839550 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.839552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.840388 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.840389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.841366 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.841370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.841922 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.841923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.841995 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.842588 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.842599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.844619 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.846692 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.846697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.846699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.847080 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.847338 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.847339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.847339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.847418 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.847749 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.847755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.847825 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.847899 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.848044 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":149,"threads_started":1}
I20260619 02:54:13.848156 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.838261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.26.98.62:39853)
0619 02:54:13.838354 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.838358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.838762 (+ 404us) server_negotiation.cc:407] Connection header received
0619 02:54:13.838841 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.838843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.838864 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.839021 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.839714 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.840285 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.841496 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.841789 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.841880 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.844762 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.844764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.844765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.844775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.846591 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.847181 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.847182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.847182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.847247 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.847521 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.847522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.847523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.847683 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.847750 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.847965 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.848054 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:54:13.848726 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.62@KRBTEST.COM
I20260619 02:54:13.856843 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.849143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.26.98.61:39291)
0619 02:54:13.849219 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.849222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.849231 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:54:13.849263 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.849264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.849277 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.849364 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.849716 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.850252 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.851206 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.851559 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.851777 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.854220 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.854222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.854222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.854233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.855701 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.856082 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.856083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.856084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.856146 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.856309 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.856310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.856310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.856444 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.856512 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.856637 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.856727 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:54:13.856843 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.849033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.61:39291 (local address 127.0.0.1:37190)
0619 02:54:13.849083 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.849095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.849138 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.849383 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.849384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.849523 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.849610 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.849613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.850372 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.850374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.851089 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.851092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.851789 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.851791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.851873 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.852546 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.852556 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.854106 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.855819 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.855820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.855821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.855999 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.856203 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.856203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.856204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.856257 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.856497 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.856498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.856551 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.856609 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.856727 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:54:13.857270 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:54:13.858251 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.848273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.26.98.62:39853)
0619 02:54:13.848546 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.848547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.848554 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:54:13.848573 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.848573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.848585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.848661 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.849552 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.850511 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.851788 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.852126 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.852202 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.854788 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.854790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.854791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.854799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.856561 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.857149 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.857151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.857151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.857226 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.857421 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.857423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.857423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.857583 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.857793 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.857982 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.858097 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":199,"threads_started":1}
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.60@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.98.61@KRBTEST.COM
I20260619 02:54:13.864780 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.857662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.26.98.60:40753)
0619 02:54:13.857701 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.857703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.857712 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:54:13.857770 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.857771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.857781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.857835 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.858121 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.858849 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.859679 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.859901 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.859957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.862164 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.862166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.862167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.862173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.863600 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.864024 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.864026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.864026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.864082 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.864245 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.864246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.864246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.864366 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.864422 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.864545 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.864653 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":23}
I20260619 02:54:13.864791 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.857599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35580)
0619 02:54:13.857679 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.857689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.857729 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.857848 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.857849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.857990 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.858039 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.858041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.859002 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.859003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.859598 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.859600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.859965 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.859966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.860020 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.860622 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.860626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.862063 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.863705 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.863706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.863707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.863917 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.864137 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.864137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.864138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.864195 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.864418 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.864419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.864458 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.864520 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.864710 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":89}
I20260619 02:54:13.867074 17376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: Bootstrap starting.
I20260619 02:54:13.867211 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.858985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.26.98.61:39291)
0619 02:54:13.859037 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.859039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.859047 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:54:13.859070 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.859071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.859081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.859144 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.859596 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.860163 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.860982 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.861281 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.861339 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.864375 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.864378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.864378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.864388 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.865986 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.866322 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.866323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.866323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.866426 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.866592 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.866593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.866594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.866746 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.866995 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.867042 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.867120 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:54:13.868134 17376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:13.868441 17376 log.cc:824] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:13.869161 27016 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:54:13.869310 17376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: No bootstrap required, opened a new log
I20260619 02:54:13.870131 17200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5617d6582b414c62bf0ae970791d238f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55444
I20260619 02:54:13.871245 17376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.871404 17376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d054782c8624902b282e42ba369f3fb, State: Initialized, Role: FOLLOWER
I20260619 02:54:13.871991 17376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.872401 17383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.872488 17376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:13.872515 17383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:54:13.875141 17394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:13.875226 17394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:13.875552 17308 master_runner.cc:186] Error getting master registration for 127.26.98.62:39853: OK, instance_id { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" instance_seqno: 1781837653437465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:54:13.879431 17308 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:54:13.885895 17301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 3
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.61@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:54:13.892639 17226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } attempt: 3
I20260619 02:54:13.897076 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.886591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.26.98.60:40753)
0619 02:54:13.886645 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.886647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.886661 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:54:13.886760 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.886762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.886775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.886842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.887340 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.888377 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.889662 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.890031 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.890130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.893241 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.893244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.893245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.893256 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.895258 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.895960 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.895963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.895964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.896044 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.896248 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.896249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.896249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.896485 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.896568 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.896848 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.896968 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:54:13.897080 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.886472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35582)
0619 02:54:13.886571 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.886591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.886669 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.886859 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.886860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.887143 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.887227 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.887230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.888561 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.888564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.889497 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.889501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.890135 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.890137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.890344 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.891122 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.891129 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.893098 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.895493 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.895496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.895496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.895817 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.896112 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.896113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.896114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.896171 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.896581 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.896584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.896669 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.896768 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.896951 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
Jun 19 02:54:13 dist-test-slave-0f9d krb5kdc[17145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837653, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.98.62@KRBTEST.COM for kudu/127.26.98.60@KRBTEST.COM
I20260619 02:54:13.899734 17301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: Bootstrap starting.
I20260619 02:54:13.900812 17301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:13.901202 17301 log.cc:824] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:13.902689 17301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: No bootstrap required, opened a new log
I20260619 02:54:13.903973 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.893061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.98.60:40753 (local address 127.0.0.1:35588)
0619 02:54:13.893134 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 02:54:13.893146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:54:13.893191 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:54:13.893835 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:54:13.893836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:54:13.894071 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:54:13.894145 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.894149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.895680 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.895682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.896267 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:54:13.896269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.896822 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.896823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:54:13.896890 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.897842 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:54:13.897848 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:54:13.899615 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:54:13.902338 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.902340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.902341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.902577 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.903010 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:54:13.903011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:54:13.903011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:54:13.903098 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:54:13.903515 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:54:13.903517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:54:13.903598 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 02:54:13.903685 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:54:13.903854 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:54:13.904287 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.893267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.26.98.60:40753)
0619 02:54:13.893572 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.893574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.893582 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:54:13.893605 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.893605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.893619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.893705 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.894452 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.895583 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.896374 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.896723 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.897679 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.899748 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.899750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.899751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.899761 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.902193 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.902661 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.902663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.902663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.902765 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.903167 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.903168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.903169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.903392 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.903515 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.903778 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.903937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":183,"threads_started":1}
I20260619 02:54:13.905448 17301 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.905666 17301 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29f8f8d72474603b9de0e7a79870937, State: Initialized, Role: FOLLOWER
I20260619 02:54:13.905771 17301 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.906270 17301 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:13.906533 17226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: Bootstrap starting.
I20260619 02:54:13.907545 17226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: Neither blocks nor log segments found. Creating new log.
I20260619 02:54:13.908274 17226 log.cc:824] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:54:13.908918 17398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.909042 17398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.909946 17226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: No bootstrap required, opened a new log
I20260619 02:54:13.911808 17226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.912036 17226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5617d6582b414c62bf0ae970791d238f, State: Initialized, Role: FOLLOWER
I20260619 02:54:13.912158 17226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.909022 17233 master_runner.cc:186] Error getting master registration for 127.26.98.62:39853: OK, instance_id { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" instance_seqno: 1781837653437465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:54:13.912585 17233 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:54:13.912606 17226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:54:13.912631 17200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:54:13.912710 17200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:54:13.913506 17409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:54:13.913698 17409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:54:13.914283 17200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.915098 17200 leader_election.cc:288] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [CANDIDATE]: Term 1 election: Requested vote from peers b29f8f8d72474603b9de0e7a79870937 (127.26.98.61:39291), 2d054782c8624902b282e42ba369f3fb (127.26.98.60:40753)
I20260619 02:54:13.915611 17275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5617d6582b414c62bf0ae970791d238f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b29f8f8d72474603b9de0e7a79870937"
I20260619 02:54:13.915753 17275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:13.915741 17411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.915884 17411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.917197 17275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5617d6582b414c62bf0ae970791d238f in term 1.
I20260619 02:54:13.917553 17350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5617d6582b414c62bf0ae970791d238f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d054782c8624902b282e42ba369f3fb"
I20260619 02:54:13.917711 17350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:54:13.918972 17350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5617d6582b414c62bf0ae970791d238f in term 1.
I20260619 02:54:13.919764 17177 leader_election.cc:302] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d054782c8624902b282e42ba369f3fb, 5617d6582b414c62bf0ae970791d238f; no voters:
I20260619 02:54:13.919958 17411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:54:13.920150 17411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [term 1 LEADER]: Becoming Leader. State: Replica: 5617d6582b414c62bf0ae970791d238f, State: Running, Role: LEADER
I20260619 02:54:13.920264 17411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } }
I20260619 02:54:13.920855 17411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5617d6582b414c62bf0ae970791d238f. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.920948 17411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:13.924028 17158 master_runner.cc:263] No leader master found from master 5617d6582b414c62bf0ae970791d238f
I20260619 02:54:13.924113 17158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:54:13.924978 17433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:54:13.925735 17433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:54:13.931623 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.916782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.26.98.62:39853)
0619 02:54:13.916834 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.916836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.919357 (+ 2521us) server_negotiation.cc:407] Connection header received
0619 02:54:13.921053 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.921055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.921069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.921101 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.921908 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.922727 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.925696 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.926050 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.926096 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.928345 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.928347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.928347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.928356 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.930157 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.930609 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.930611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.930611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.930675 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.930869 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.930870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.930871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.931085 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.931177 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.931399 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.931510 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 02:54:13.932582 17179 tablet.cc:2369] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:54:13.933279 17350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [term 1 FOLLOWER]: Refusing update from remote peer 5617d6582b414c62bf0ae970791d238f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:54:13.933681 17411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:54:13.936187 17383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5617d6582b414c62bf0ae970791d238f. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.936275 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.917396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.26.98.61:39291)
0619 02:54:13.917508 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.917524 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.919474 (+ 1950us) server_negotiation.cc:407] Connection header received
0619 02:54:13.919592 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.919593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.919608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.919640 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.923903 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.924624 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.926545 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.926873 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.926940 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.932930 (+ 5990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.932933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.932933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.932944 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.934768 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.935290 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.935293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.935293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.935376 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.935592 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.935593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.935593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.935746 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.935871 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.936084 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.936181 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 02:54:13.936314 17383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.937943 17275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [term 1 FOLLOWER]: Refusing update from remote peer 5617d6582b414c62bf0ae970791d238f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:54:13.938513 17411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:54:13.940037 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:13.917461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.26.98.60:40753)
0619 02:54:13.917561 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:13.917563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:13.932108 (+ 14545us) server_negotiation.cc:407] Connection header received
0619 02:54:13.932222 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:13.932224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:13.932239 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:13.932270 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:13.932888 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.933686 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.934738 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:13.935083 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:13.935131 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:13.936620 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:54:13.936622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:54:13.936622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:54:13.936631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:54:13.938412 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.938959 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.938960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.938961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.939037 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:54:13.939241 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:54:13.939242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:54:13.939243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:54:13.939421 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:54:13.939545 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:13.939775 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 02:54:13.939918 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:54:13.940683 17398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5617d6582b414c62bf0ae970791d238f. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.940821 17398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.942396 17411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.942528 17411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:13.942559 17398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.942674 17398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.943141 17435 mvcc.cc:205] Tried to move back new op lower bound from 7298407030502600704 to 7298407030458155008. Current Snapshot: MvccSnapshot[applied={T|T < 7298407030502600704}]
I20260619 02:54:13.948468 17438 mvcc.cc:205] Tried to move back new op lower bound from 7298407030502600704 to 7298407030458155008. Current Snapshot: MvccSnapshot[applied={T|T < 7298407030502600704}]
I20260619 02:54:13.949872 17431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.950021 17431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f [sys.catalog]: This master's current role is: LEADER
I20260619 02:54:13.950721 17383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5617d6582b414c62bf0ae970791d238f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5617d6582b414c62bf0ae970791d238f" member_type: VOTER last_known_addr { host: "127.26.98.62" port: 39853 } } peers { permanent_uuid: "b29f8f8d72474603b9de0e7a79870937" member_type: VOTER last_known_addr { host: "127.26.98.61" port: 39291 } } peers { permanent_uuid: "2d054782c8624902b282e42ba369f3fb" member_type: VOTER last_known_addr { host: "127.26.98.60" port: 40753 } } }
I20260619 02:54:13.950856 17383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:54:13.951043 17433 catalog_manager.cc:1347] Generated new cluster ID: c80f9a3a516a4afba09a50269216adc1
I20260619 02:54:13.951097 17433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:54:13.951264 17436 mvcc.cc:205] Tried to move back new op lower bound from 7298407030502600704 to 7298407030458155008. Current Snapshot: MvccSnapshot[applied={T|T < 7298407030502600704}]
I20260619 02:54:13.989446 17433 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:54:13.990914 17433 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:54:14.001869 17433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5617d6582b414c62bf0ae970791d238f: Generated new TSK 0
I20260619 02:54:14.002240 17433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:54:14.032384 17381 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.030295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.26.98.62:39853)
0619 02:54:14.030360 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.030362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.031469 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 02:54:14.031599 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.031600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.031615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.031651 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.032286 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55466: BlockingRecv error: recv got EOF from 127.0.0.1:55466 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 02:54:14.033311 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.030823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.26.98.61:39291)
0619 02:54:14.030871 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.030873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.032725 (+ 1852us) server_negotiation.cc:407] Connection header received
0619 02:54:14.032825 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.032826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.032840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.032871 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.033234 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37226: BlockingRecv error: recv got EOF from 127.0.0.1:37226 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 02:54:14.034164 17396 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.031320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.26.98.60:40753)
0619 02:54:14.031376 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.031378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.033589 (+ 2211us) server_negotiation.cc:407] Connection header received
0619 02:54:14.033692 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.033694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.033708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.033738 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.034088 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35598: BlockingRecv error: recv got EOF from 127.0.0.1:35598 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 02:54:14.041191 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.039971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.26.98.61:39291)
0619 02:54:14.040032 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.040034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.040380 (+ 346us) server_negotiation.cc:407] Connection header received
0619 02:54:14.040513 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.040516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.040540 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.040573 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.041110 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37240: BlockingRecv error: recv got EOF from 127.0.0.1:37240 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 02:54:14.045514 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.044271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.26.98.61:39291)
0619 02:54:14.044325 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.044328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.044693 (+ 365us) server_negotiation.cc:407] Connection header received
0619 02:54:14.044818 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.044819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.044834 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.044867 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.045428 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37256: BlockingRecv error: recv got EOF from 127.0.0.1:37256 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 02:54:14.050509 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.049317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.26.98.61:39291)
0619 02:54:14.049378 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.049380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.049744 (+ 364us) server_negotiation.cc:407] Connection header received
0619 02:54:14.049866 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.049868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.049882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.049915 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.050429 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37266: BlockingRecv error: recv got EOF from 127.0.0.1:37266 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 02:54:14.057571 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.056363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.26.98.61:39291)
0619 02:54:14.056420 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.056422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.056809 (+ 387us) server_negotiation.cc:407] Connection header received
0619 02:54:14.056930 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.056931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.056946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.056979 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.057489 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37268: BlockingRecv error: recv got EOF from 127.0.0.1:37268 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 02:54:14.068620 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.067387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.26.98.61:39291)
0619 02:54:14.067448 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.067451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.067832 (+ 381us) server_negotiation.cc:407] Connection header received
0619 02:54:14.067949 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.067951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.067966 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.068000 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.068535 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37270: BlockingRecv error: recv got EOF from 127.0.0.1:37270 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 02:54:14.087524 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.086299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.26.98.61:39291)
0619 02:54:14.086362 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.086365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.086742 (+ 377us) server_negotiation.cc:407] Connection header received
0619 02:54:14.086858 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.086859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.086874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.086943 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.087444 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37280: BlockingRecv error: recv got EOF from 127.0.0.1:37280 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 02:54:14.122759 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.121707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.26.98.61:39291)
0619 02:54:14.121768 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.121770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.122032 (+ 262us) server_negotiation.cc:407] Connection header received
0619 02:54:14.122079 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.122080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.122097 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.122176 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.122681 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37284: BlockingRecv error: recv got EOF from 127.0.0.1:37284 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 02:54:14.189786 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.188491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.26.98.61:39291)
0619 02:54:14.188581 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.188583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.188637 (+ 54us) server_negotiation.cc:407] Connection header received
0619 02:54:14.188913 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.188915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.188928 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.189016 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.189709 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37290: BlockingRecv error: recv got EOF from 127.0.0.1:37290 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 02:54:14.320411 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.319282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.26.98.61:39291)
0619 02:54:14.319438 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.319449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.319538 (+ 89us) server_negotiation.cc:407] Connection header received
0619 02:54:14.319735 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.319737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.319756 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.319828 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.320261 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37300: BlockingRecv error: recv got EOF from 127.0.0.1:37300 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260619 02:54:14.578959 17306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:54:14.578149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.26.98.61:39291)
0619 02:54:14.578220 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:14.578222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:14.578237 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:54:14.578343 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:14.578344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:14.578355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:14.578396 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:54:14.578845 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57036: BlockingRecv error: recv got EOF from 127.0.0.1:57036 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:54:14.875782 17394 catalog_manager.cc:1259] Loaded cluster ID: c80f9a3a516a4afba09a50269216adc1
I20260619 02:54:14.875916 17394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: loading cluster ID for follower catalog manager: success
I20260619 02:54:14.876852 17394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: acquiring CA information for follower catalog manager: success
I20260619 02:54:14.877231 17394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d054782c8624902b282e42ba369f3fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:54:14.914423 17409 catalog_manager.cc:1259] Loaded cluster ID: c80f9a3a516a4afba09a50269216adc1
I20260619 02:54:14.914525 17409 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: loading cluster ID for follower catalog manager: success
I20260619 02:54:14.915652 17409 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: acquiring CA information for follower catalog manager: success
I20260619 02:54:14.916088 17409 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b29f8f8d72474603b9de0e7a79870937: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:54:15.097642 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:15.092689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.26.98.61:39291)
0619 02:54:15.092829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:15.092831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:15.093048 (+ 217us) server_negotiation.cc:407] Connection header received
0619 02:54:15.093090 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:15.093091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:15.093165 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:15.093418 (+ 253us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:15.093777 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.094681 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.095949 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.096210 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.096397 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:15.096651 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:15.096816 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:15.096901 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:15.097149 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 02:54:15.097188 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260619 02:54:15.103952 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:15.100622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.26.98.60:40753)
0619 02:54:15.100757 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:15.100758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:15.100769 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:54:15.100822 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:15.100822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:15.100874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:15.100969 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:15.101198 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.101844 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.102957 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.103186 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.103256 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:15.103510 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:15.103665 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:15.103741 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:15.103804 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:54:15.103832 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260619 02:54:15.112144 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:15.106688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.26.98.61:39291)
0619 02:54:15.106738 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:15.106740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:15.106818 (+ 78us) server_negotiation.cc:407] Connection header received
0619 02:54:15.106992 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:15.106993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:15.107010 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:15.107071 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:15.107294 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.108103 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.109927 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.110407 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.110516 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:15.110806 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:15.111382 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:15.111471 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:15.112009 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 02:54:15.112063 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:54:15.112509 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:15.106909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.26.98.60:40753)
0619 02:54:15.106972 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:15.106974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:15.107163 (+ 189us) server_negotiation.cc:407] Connection header received
0619 02:54:15.107255 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:15.107256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:15.107269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:15.107319 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:15.107493 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.108250 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.110585 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.111397 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.111766 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:15.111951 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:15.111997 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:15.112148 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:15.112341 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:54:15.112418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:54:15.112746 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:54:15.106609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.26.98.62:39853)
0619 02:54:15.106975 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 02:54:15.106976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:54:15.106984 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:54:15.107074 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:54:15.107075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:54:15.107158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:54:15.107286 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:54:15.108378 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.108914 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.110636 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:54:15.111494 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:54:15.111830 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:54:15.112170 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:54:15.112368 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:54:15.112606 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 02:54:15.112625 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:54:15.112657 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":73,"threads_started":1}
I20260619 02:54:15.116107 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 17158
I20260619 02:54:15.123128 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 17233
I20260619 02:54:15.128928 27016 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcBfLLw/build/release/bin/kudu with pid 17308
2026-06-19T02:54:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3027 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6023 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302497 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST